EBCDIC code pages and mappings files

IBM Z® Open Editor and RSE API Plug-in for Zowe™ CLI (RSE CLI plug-in) can handle EBCDIC code page conversion mappings for MVS and UNIX System Services files. You can set an encoding default in the RSE profile, use an encoding tag with Upload and Download MVS commands, use .gitattributes files for uploading and downloading files from UNIX System Services, and use mappings files either provided by the RSE API host component or used at a project level. See this page with the list of all the supported code pages.

Zowe capabilities

When you work with z/OS® resources through Zowe Explorer or Zowe CLI, character conversion between UTF-8 (the default encoding used by VS Code and the only local encoding for which correct conversion is currently guaranteed) and EBCDIC code pages is handled automatically by the z/OSMF and RSE API servers. Files stored on z/OS in EBCDIC are converted to UTF-8 when downloaded for display and editing in VS Code, and UTF-8 files are converted back to EBCDIC when uploaded to z/OS. To ensure accurate conversion, you must configure which EBCDIC code page corresponds to each z/OS resource. The following sections describe the various methods for specifying code page mappings for use with Zowe Explorer or Zowe CLI.

Zowe CLI

Many commands for Zowe CLI as well as the IBM RSE API Plug-in for Zowe CLI have additional parameters for specifying a code page directly or specifying a mappings file to be used for computing the code page encoding.

For example, assuming the member SAM1 was written and stored on MVS using the IBM-273 EBCDIC encoding, you can work with it via the following commands.

• To download the member SAM1 as a UTF-8 file (converting from IBM-273), issue the following command:

  zowe rse download ds "IBMUSER.SAMPLE.COBOL(SAM1)" --ec IBM-273
  

• To upload the member SAM1 using the IBM-273 EBCDIC encoding, issue the following command:

   zowe rse upload ftds ./ibmuser/sample/cobol/SAM1.txt "IBMUSER.SAMPLE.COBOL(SAM1)" --ec IBM-273
  

Zowe Profile default encoding

You can also provide a default encoding value to be used for your Zowe CLI commands as well as Zowe Explorer operations in your Zowe profiles themselves. To do that, complete the following steps:

1. Create or open your Zowe team configuration file.

2. Insert the encoding property under the "properties" object of the relevant profile.

Following is an example of how to do this:

{
  "profiles": {
    "rse": {
      "type": "rse",
      "properties": {
        "port": 6802,
        "basePath": "rseapi",
        "protocol": "https",
        "encoding": "IBM-930"
      },
      "secure": ["user", "password"]
    }
  }
}

Advanced IBM RSE API capabilities

The RSE API Server provides the capability to manage mappings files that allow defining how MVS files should be converted when uploaded and downloaded. These mappings do not cover z/OS UNIX System Services, but see Character conversion using .gitattributes instead.

The RSE CLI plug-in provides the ability to download a default mappings file from the server. This file is maintained by the RSE API server administrator and can be retrieved using the RSE CLI plug-in command zowe rse check conversion-mappings. You can modify this file in the ~/.zowe/profiles/rse folder, and the tools automatically use the updated mappings file when they interact with the server. The mapping behavior is consistent with similar functionality offered in IBM Developer for z/OS through its user interface.

To download the default mappings file provided by your RSE API server administrator, issue the following command:

zowe rse check conversion-mappings

The downloaded mappings file will be placed in ~/.zowe/profiles/rse/conversion-mappings.json. You can add your personal conversion preferences to this file and the updates will be automatically used by all your RSE CLI plug-in commands without the need to specify a command line parameter.

However, if you manage more than one such file, for example for different projects, or you even prefer defining such mappings in a ZAPP file, then you can specify the file to use with a command line parameter as well:

zowe rse upload ftds ./ibmuser/sample/cobol/SAM1.txt "IBMUSER.SAMPLE.COBOL(SAM1)" --maps ./zapp.yaml

In addition to mappings files that are stored in the user's .zowe directory and used by the RSE CLI plug-in, ZAPP provides the same ability of defining mappings using the same schema, but offers YAML as additional syntax. This allows teams to share these mappings for their applications and make them part of the source code that could be stored in a Git repository or other SCM. ZAPP files can either be in YAML or JSON syntax; however, it is recommended to have your ZAPP file in one syntax only, not both, within the same project or directory.

MVS encoding precedence rules

Tools such as the RSE CLI plug-in or Z Open Editor find mappings following these precedence rules with some rules applied to CLI only:

1. (CLI only) An encoding | ec parameter provided by the command should always have highest precedence, for example --encoding IBM-273.
2. (CLI only) An encoding found in a file that was provided via the parameter mappings-file | maps, for example --mappings-file ~/myproject/zapp.yaml.
3. An encoding found in the current workspace in a file named zapp.yaml or zapp.json.
4. An encoding found in the default mappings file in the user's ~/.zowe/profiles/rse folder that was downloaded via command line.
5. An encoding specified in the Zowe CLI profile itself.
6. No encoding is specified, resulting in the use of the server default mappings file residing on the server.

Mappings file schema

The JSON schema for defining RSE CLI plug-in profiles simply consists of an array of mappings and a default encoding to be used if none of the mappings provided apply. The mappings can be provided as a simple array called mappings of mappings objects in a JSON file or as part of a ZAPP profile with settings as shown in the diagram below. See examples further below.

Fig.1: ZAPP Schema Model for RSE API

When reading these mappings the following validation and interpretation rules apply:

• There can only be one ZAPP rse-mappings profile of type rseapi in a ZAPP file.
• Member mappings consist of the same properties as the mappings except for the member mappings itself, that is, only one level of nesting.
• When member mappings are matched, they will override values of the parent mappings. For example, the parent data set mapping might define a default encoding that is true for members that do not match, but if a member mapping is valid, any encoding value specified there will override the parent mapping.
• The resource property is a mandatory property for data set and member mappings. Note: Mapping against members in all data sets is not possible for performance reasons.
• The resource of the data set parent represents a pattern on the data set name.
• The resource of the member mapping represents a pattern on the member name.
• If no transfer is specified for a resource mapping, the default is text.
• If no encoding is defined for text, it will use the default specified in the local mappings file. If that is missing, it will use the default of the server.
• If a transfer is specified as binary, the encoding will be ignored.
• The extension property is only used for downloading files to determine which extension to append to the member name, which is to ensure editors can open the file correctly. For uploading files, only the mappings to resources are relevant.
• An extension entry can be present either in a member mapping or the data set mapping. If it is in both mappings, the member mapping takes precedence. If it is missing completely in mappings, the first resource match will be used.

Resources can be specified following the rules defined in Step 5 of the z/OS Explorer docs for Mapping data sets and partitioned data set members.

There are two wildcards allowed: ** and *

• ** matches in data set names across multiple name hierarchies. It cannot be used for member names. For example: **ERROR** matches:
  • USER.ERROR.TEST
  • USER.A.B.MYERROR.ABC
  • USER.A.B.MYERROR2.ABC
• * matches within a data set name segment or the member name. For example, USER**COB* matches:
  • USER.A.COB
  • USER.A.B.COBOL
  • USER.A.B.ACOBCOPY

Example JSON mappings file

The following shows a sample mappings file using JSON that you could place in your ~/.zowe/profiles/rse as conversion-mappings.json:

{
  "mappings": [
    {
      "resource": "**.SAMPLE.COB**",
      "extension": "cbl",
      "transfer": "text",
      "encoding": "IBM-273",
      "memberMappings": [
        {
          "extension": "dat",
          "transfer": "binary",
          "resource": "EIC*"
        }
      ]
    },
    {
      "resource": "**PLI*",
      "extension": "pl1",
      "transfer": "text",
      "encoding": "IBM-500"
    }
  ],
  "default.encoding": "IBM-1047"
}

This file defines that data sets containing the string .SAMPLE.COB in their fully qualified name, such as USER1.SAMPLE.COBOL, are transferred as text with the IBM-273 EBCDIC code page (Austria/Germany) on MVS. When you download a data set member with an RSE CLI plug-in command or Zowe Explorer using an RSE profile, RSE API converts the content from IBM-273 EBCDIC to UTF-8, the format required by Z Open Editor. Uploading such a file converts the UTF-8 back to IBM-273 EBCDIC. The extension property, currently supported only by the CLI, specifies that the downloaded data set member is stored as a file with a .cbl file extension.

In addition to the default rule for **.SAMPLE.COB** data sets, a nested memberMappings subrule provides a refinement: If the member name contains EIC as a substring, the file is not converted from IBM-273 but treated as binary instead. This enables data sets to contain members of different types.

The example shows how members in data sets with names containing PLI are converted using IBM-500 (Belgium/Canada/Switzerland). In all other cases, the default conversion code page is IBM-1047.

Example ZAPP file

In a zapp.yaml file, you can specify the same mapping described in the previous section by defining a ZAPP profile of type rseapi. The zapp.yaml for this particular specification would contain text as follows:

name: sam
version: 1.1.0

profiles:
  - name: RSE-Mappings
    type: rseapi
    settings:
      mappings:
        - resource: "**.SAMPLE.COB**"
          extension: cbl
          transfer: text
          encoding: IBM-273
          memberMappings:
            - extension: dat
              transfer: binary
              resource: "*DAT"
        - resource: "**PLI*"
          extension: pl1
          transfer: text
          encoding: IBM-500
      default.encoding: IBM-1047

Using mappings files with Z Open Editor

Mappings files are not only used by RSE CLI plug-in commands, but also by Z Open Editor remote include file resolution operations. When you define a property group either in your ZAPP file or Settings, the retrieval of include files or copybooks will use mappings files based on the precedence rules defined above. For example, if you define mappings in a zapp.yaml file in your workspace, these mappings will be used. If you do not have a ZAPP file, but have a personal mappings file in ~/.zowe/profiles/rse/conversion-mappings.json, the personal file will be automatically used.

Using mappings files with Zowe Explorer

Zowe Explorer enables you to navigate your data sets and open data set members in the editor with a simple click. When you use Zowe Explorer with an RSE profile, conversion mappings are applied as well. To enable this, specify the absolute location of the mappings file in your user settings. The setting is called zopeneditor.zowe.defaultRseConversionMappingsFile. You can find it in the IBM Z Open Editor settings together with the other Zowe-related settings, such as the defaultCliProfile. Conversion mappings defined in either a JSON-formatted RSE API user mappings file or a ZAPP file override an encoding specified in a Zowe profile.

RSE CLI plug-in logging

When using encoding and mappings files with the RSE CLI plug-in commands, logging will be written and found at ~/.zowe/zowe/logs/zowe.log. This log file will contain details about the encoding precedence that was used during upload and download of MVS files, the location of the mappings file if available, and the outcome of the encoding or transfer type that was chosen according to the previously-mentioned values.

In addition to the logging, you can also refer to the Upload and Download output to troubleshoot. If binary transfer was used, the output includes a binary value of true. If the file was transferred as text and an encoding value was used, the output includes an encoding value for each file.

Specifying an EBCDIC file encoding to use with the language servers

While the previous sections covered character conversion during file transfer between your local workspace and z/OS resources, the language servers operate entirely within Z Open Editor. You can specify an EBCDIC code page for the language servers to use when they parse your programs. This enables the language servers to parse programs in their source code page, ensuring accurate syntax validation and line-length calculations. The specified encoding must match the code page used when the file is stored on z/OS. This functionality is available for the COBOL, PL/I, and JCL language servers:

1. For local COBOL, PL/I, or JCL programs, configure the zopeneditor.encodings.filePatterns settings object.

   • This setting maps file name glob patterns to an EBCDIC file encoding. These patterns are specified as glob patterns. A single asterisk * matches any number of characters within a path segment, including none. A double asterisk ** matches any number of path segments, including none. For example, { **/COBOL/*.cbl : IBM-930 } assigns any file paths that contain a directory named COBOL and end with a .cbl extension an encoding of IBM-930. (Learn more about how to specify glob patterns.)

2. For remote programs (accessed through Zowe Explorer), you can do any of the following:

   1. If an RSE API profile is used to access the file (such as a data set opened in Zowe Explorer), Z Open Editor first looks for an encoding specified by your RSE mappings as documented above (either in a rseapi profile in your zapp.yaml or the location in the zopeneditor.zowe.defaultRseConversionMappingsFile setting).
   2. If no RSE mappings exist or if you are using a z/OSMF profile to access the file, you can specify an encoding in the Zowe Explorer profile you are using to access the program (see the encoding property example in this section).
   3. You can use the zopeneditor.encodings.filePatterns user setting for remote files, but you must adjust the glob pattern to match to remote files in the Zowe Explorer filesystem (described here).

These settings only control what encodings are passed to the language servers. To control the encodings used for file conversions when you upload or download z/OS resources, use an RSE API mappings file, a rseapi ZAPP profile, or specify the encoding property in your Zowe profile.

When a program is opened with an encoding specified using one of the preceding methods, a counter titled "EBCDIC Byte Count" appears in the VS Code status bar. This counter displays two numbers:

• The number of bytes in the line at the cursor's current position (accounting for the source code page).
• The maximum number of bytes enabled in the line (controlled by the zopeneditor.language.maximumLineLength setting). As with the previous functionality, this counter displays for COBOL, PL/I, and JCL programs.

If the language server detects double-byte characters, it displays guillemets (»«) at the positions of the SOSI codes, when applicable. These guillemets are decorative; they are not stored with the program and do not affect line length. The byte counter, along with SOSI position detection and decoration, is supported for COBOL, PL/I, and JCL programs that are assigned an EBCDIC file encoding.

Character conversion using .gitattributes

The .gitattributes file allows you to specify how files should be encoded when they are checked out to the working directory and when they are stored in the Git repository. This is particularly useful for handling double-byte characters.

How it works

1. User build from VS Code: When you perform a user build on an existing COBOL file that contains double-byte characters, the .gitattributes settings ensure that the file is encoded in IBM-930 when it is checked out to the working directory.
2. Build process: During the build process, the file is uploaded to the specified UNIX System Services folder with the encoding specified in .gitattributes.
3. File display in VS Code: When you open the uploaded files in VS Code through the [z/OS Resources Table](./advanced_resource_table.md or Zowe Explorer), the double-byte characters are converted from IBM-930 to UTF-8 during download for display in the text editor. The source file on z/OS remains unmodified.

Example workflow

1. Update the .gitattributes file located at the root of the repository with the following line. This specifies an encoding of IBM-930 for all files in the repository with the .cbl file extension: *.cbl zos-working-tree-encoding=ibm-930 git-encoding=utf-8.
2. Perform a user build from VS Code on an existing COBOL file that contains double-byte characters.
3. During the build process, the zos-working-tree-encoding parameter specifies the code page used to encode files that are uploaded to UNIX System Services or MVS during the build.
4. Wait for the build process to complete. Once finished, the COBOL file should be uploaded to the specified USS folder using the encoding from .gitattributes.