EBCDIC code pages and mappings files

IBM Z Open Editor and RSE API Plug-in for Zowe CLI (RSE CLI plug-in) can handle EBCDIC code page conversion mappings for MVS and UNIX System Services files. You can set an encoding default in the RSE profile, use an encoding tag with Upload and Download MVS commands, use .gitattributes files for uploading and downloading files from UNIX System Services, and use mappings files either provided by the RSE API host component or used at a project level. See this page with the list of all the supported code pages.

Zowe capabilities

Many commands for Zowe CLI as well as the IBM RSE API Plug-in for Zowe CLI have additional parameters for specifying a code page directly or specifying a mappings file to be used for computing the code page encoding.

For example, assuming the member SAM1 was written and stored on MVS using the IBM-273 EBCDIC encoding, you can work with it via the following commands.

• To download the member SAM1 as a UTF-8 file (converting from IBM-273), issue the following command:

  zowe rse download ds "IBMUSER.SAMPLE.COBOL(SAM1)" --ec IBM-273
  

• To upload the member SAM1 using the IBM-273 EBCDIC encoding, issue the following command:

   zowe rse upload ftds ./ibmuser/sample/cobol/SAM1.txt "IBMUSER.SAMPLE.COBOL(SAM1)" --ec IBM-273
  

You can also provide a default encoding value to be used for your Zowe CLI commands as well as Zowe Explorer operations in your Zowe profiles themselves. To do that, complete the following steps:

1. Create or locate your Zowe team configuration file.

2. Insert the encoding property under the "properties" object of the relevant profile.

Following is an example of how to do this:

{
  "profiles": {
    "rse": {
      "type": "rse",
      "properties": {
        "port": 6802,
        "basePath": "rseapi",
        "protocol": "https",
        "encoding": "IBM-930"
      },
      "secure": ["user", "password"]
    }
  }
}

When uploading a file via Zowe Explorer - Zowe Profile:

1. The file will be converted to the encoding that is specified in the selected Zowe profile.
2. The converted file is saved on the mainframe with the specified encoding.
3. When you download the file, it will be converted back from the specified encoding to UTF-8.

This ensures that double-byte characters such as Japanese characters are correctly handled and displayed.

Example workflow:

1. Upload a file that contains double-byte characters by using Zowe Explorer and save it on the mainframe.
2. Use the uploaded file as a copybook reference in another file and click on the reference.
3. The referenced file should open in a separate window in VS Code, displaying that the Japanese characters are correctly converted from IBM-930 to UTF-8.

Advanced IBM RSE API capabilities

The RSE API Server provides the capability to manage mappings files that allow defining how MVS files should be converted when uploaded and downloaded. These mappings do not cover z/OS UNIX System Services, but see Character conversion using .gitattributes instead.

RSE CLI plug-in provides the ability to get a default mappings file, which is maintained by the RSE API server administrator, from the server that can be downloaded using the RSE CLI plug-in command zowe rse check conversion-mappings. You can make personal modifications to this file in the ~/.zowe/profiles/rse folder, and then have our tools use this mappings file automatically to interact with the server. The mapping is consistent with a similar behavior offered in IBM Developer for z/OS (IDz) via its user interface.

To download the default mappings file provided by your RSE API server administrator, issue the following command:

zowe rse check conversion-mappings

The downloaded mappings file will be placed in ~/.zowe/profiles/rse/conversion-mappings.json. You can add your personal conversion preferences to this file and the updates will be automatically used by all your RSE CLI plug-in commands without the need to specify a command line parameter.

However, if you manage more than one such file, for example for different projects, or you even prefer defining such mappings in a ZAPP file, then you can specify the file to use with a command line parameter as well:

zowe rse upload ftds ./ibmuser/sample/cobol/SAM1.txt "IBMUSER.SAMPLE.COBOL(SAM1)" --maps ./zapp.yaml

In addition to mappings files that are stored in the user's .zowe directory and used by the RSE CLI plug-in, ZAPP provides the same ability of defining mappings using the same schema, but offers YAML as additional syntax. This allows teams to share these mappings for their applications and make them part of the source code that could be stored in a Git repository or other SCM. ZAPP files can either be in YAML or JSON syntax; however, it is recommended to have your ZAPP file in one syntax only, not both, within the same project or directory.

MVS encoding precedence rules

Tools such as the RSE CLI plug-in or Z Open Editor find mappings following these precedence rules with some rules applied to CLI only:

1. (CLI only) An encoding | ec parameter provided by the command should always have highest precedence, for example --encoding IBM-273.
2. (CLI only) An encoding found in a file that was provided via the parameter mappings-file | maps, for example --mappings-file ~/myproject/zapp.yaml.
3. An encoding found in the current workspace in a file named zapp.yaml or zapp.json.
4. An encoding found in the default mappings file in the user's ~/.zowe/profiles/rse folder that was downloaded via command line.
5. An encoding specified in the Zowe CLI profile itself.
6. No encoding is specified, resulting in the use of the server default mappings file residing on the server.

Mappings file schema

The JSON schema for defining RSE CLI plug-in profiles simply consists of an array of mappings and a default encoding to be used if none of the mappings provided apply. The mappings can be provided as a simple array called mappings of mappings objects in a JSON file or as part of a ZAPP profile with settings as shown in the diagram below. See examples further below.

Fig.1: ZAPP Schema Model for RSE API

When reading these mappings the following validation and interpretation rules apply:

• There can only be one ZAPP rse-mappings profile of type rseapi in a ZAPP file.
• Member mappings consist of the same properties as the mappings except for the member mappings itself, that is, only one level of nesting.
• When member mappings are matched, they will override values of the parent mappings. For example, the parent data set mapping might define a default encoding that is true for members that do not match, but if a member mapping is valid, any encoding value specified there will override the parent mapping.
• The resource property is a mandatory property for data set and member mappings. Note: Mapping against members in all data sets is not possible for performance reasons.
• The resource of the data set parent represents a pattern on the data set name.
• The resource of the member mapping represents a pattern on the member name.
• If no transfer is specified for a resource mapping, the default is text.
• If no encoding is defined for text, it will use the default specified in the local mappings file. If that is missing, it will use the default of the server.
• If a transfer is specified as binary, the encoding will be ignored.
• The extension property is only used for downloading files to determine which extension to append to the member name, which is to ensure editors can open the file correctly. For uploading files, only the mappings to resources are relevant.
• An extension entry can be present either in a member mapping or the data set mapping. If it is in both mappings, the member mapping takes precedence. If it is missing completely in mappings, the first resource match will be used.

Resources can be specified following the rules defined in Step 5 of the z/OS Explorer docs for Mapping data sets and partitioned data set members.

There are two wildcards allowed: ** and *

• ** matches in data set names across multiple name hierarchies. It cannot be used for member names. For example: **ERROR** matches:
  • USER.ERROR.TEST
  • USER.A.B.MYERROR.ABC
  • USER.A.B.MYERROR2.ABC
• * matches within a data set name segment or the member name. For example, USER**COB* matches:
  • USER.A.COB
  • USER.A.B.COBOL
  • USER.A.B.ACOBCOPY

Example JSON mappings file

The following shows a sample mappings file using JSON that you could place in your ~/.zowe/profiles/rse as conversion-mappings.json:

{
  "mappings": [
    {
      "resource": "**.SAMPLE.COB**",
      "extension": "cbl",
      "transfer": "text",
      "encoding": "IBM-273",
      "memberMappings": [
        {
          "extension": "dat",
          "transfer": "binary",
          "resource": "EIC*"
        }
      ]
    },
    {
      "resource": "**PLI*",
      "extension": "pl1",
      "transfer": "text",
      "encoding": "IBM-500"
    }
  ],
  "default.encoding": "IBM-1047"
}

This file defines that files located in data sets containing the string .SAMPLE.COB in their fully qualified name such as USER1.SAMPLE.COBOL would be transferred as text assuming an IBM-273 EBCDIC code page (Austria/Germany) on MVS. Hence, downloading a data set member with an RSE CLI plug-in command or Zowe Explorer using an RSE profile would therefore convert the content by RSE API from IBM-273 EBCDIC to a UTF-8 file, as that is the format Z Open Editor requires files to be in. Uploading such a file would convert the UTF-8 back to IBM-273 EBCDIC. The extension property, which is currently only supported by the CLI, specifies that the downloaded data set member would be stored as a file with a .cbl file extension.

In addition to the default rule for **.SAMPLE.COB** data sets, you see a nested memberMappings subrule that provides a refinement: If the member name contains EIC as a substring then the file will not be converted from IBM-273, but treated as a binary instead. This allows you to use data sets with members of different types.

Furthermore, the example shows how members in data sets that contain PLI in their names would be converted using IBM-500 (Belgium/Canada/Switzerland) as well as that in all other cases the default conversion code page would be IBM-1047.

Example ZAPP file

In a zapp.yaml file, you can specify the same mapping described in the previous section by defining a ZAPP profile of type rseapi. The zapp.yaml for this particular specification would contain text as follows:

name: sam
version: 1.1.0

profiles:
  - name: RSE-Mappings
    type: rseapi
    settings:
      mappings:
        - resource: "**.SAMPLE.COB**"
          extension: cbl
          transfer: text
          encoding: IBM-273
          memberMappings:
            - extension: dat
              transfer: binary
              resource: "*DAT"
        - resource: "**PLI*"
          extension: pl1
          transfer: text
          encoding: IBM-500
      default.encoding: IBM-1047

Using mappings files with Z Open Editor

Mappings files are not only used by RSE CLI plug-in commands, but also by Z Open Editor remote include file resolution operations. When you define a property group either in your ZAPP file or Settings, the retrieval of include files or copybooks will use mappings files based on the precedence rules defined above. For example, if you define mappings in a zapp.yaml file in your workspace, these mappings will be used. If you do not have a ZAPP file, but have a personal mappings file in ~/.zowe/profiles/rse/conversion-mappings.json, the personal file will be automatically used.

Using mappings files with Zowe Explorer

Zowe Explorer allows you to navigate your data sets and open data set members in the editor with a simple click. When you use Zowe Explorer with an RSE profile, conversion mappings can be used as well. To enable this, you need specify the absolute location of the mappings file to use in your user settings. The setting is called zopeneditor.zowe.defaultRseConversionMappingsFile. You can find it in the IBM Z Open Editor settings together with the other Zowe-related settings such as the defaultCliProfile. Conversion mappings defined in either a JSON-formatted RSE API user mappings file or a ZAPP file will override an encoding specified in a Zowe profile.

RSE CLI plug-in logging

When using encoding and mappings files with the RSE CLI plug-in commands, logging will be written and found at ~/.zowe/zowe/logs/zowe.log. This log file will contain details about the encoding precedence that was used during upload and download of MVS files, the location of the mappings file if available, and the outcome of the encoding or transfer type that was chosen according to the previously-mentioned values.

In addition to the logging, you can also refer to the Upload and Download output to troubleshoot. If binary transfer was used, the output includes a binary value of true. If the file was transferred as text and an encoding value was used, the output includes an encoding value for each file.

Character conversion using .gitattributes

The .gitattributes file allows you to specify how files should be encoded when they are checked out to the working directory and when they are stored in the Git repository. This is particularly useful for handling double-byte characters.

How it works

1. User Build from VS Code: When you perform a user build on an existing COBOL file that contains double-byte characters, the .gitattributes settings ensure that the file is encoded in IBM-930 when it is checked out to the working directory.
2. Build process: During the build process, the file is uploaded to the specified USS folder with the correct encoding that is mentioned in the .gitattributes.
3. File display in VS Code: When you open the file in VS Code, the double-byte characters are correctly converted from IBM-930 to UTF-8.

Example workflow

1. Update the .gitattributes file located at the root of the repository with the following line. This specifies an encoding of IBM-930 for all files in the repository with the .cbl file extension: *.cbl zos-working-tree-encoding=ibm-930 git-encoding=utf-8.
2. Perform a user build from VS Code on an existing COBOL file containing double-byte characters using the right-click menu.
3. During the build process, the zos-working-tree-encoding parameter specifies the code page used for encoding files that are uploaded to USS or MVS.
4. Wait for the build process to complete. Once finished, the COBOL file should be uploaded to the specified USS folder using the encoding from .gitattributes.
5. When opened in VS Code, the Japanese characters should be correctly converted from IBM-930 to UTF-8.

Example file with double-byte characters

こんにちは、世界！ //* Hello world!
これはテストファイルです。 //* This is a test file.
IBM-930エンコーディングを使用しています。 //* Using IBM-930 encoding.

Specifying an EBCDIC file encoding to use with the language servers

Our language servers parse programs using the IBM-037 code page by default. If your program uses a different target encoding on z/OS, you must specify an EBCDIC code page for the language server to use so it can compute the correct line length. This encoding should match the EBCDIC encoding used when the file is placed back on z/OS. Here is how Z Open Editor evaluates what encoding to pass to the language server:

1. If an RSE profile was used to access the file (such as a data set opened in Zowe Explorer), Z Open Editor will first look for an encoding specified by your RSE mappings (either in your zapp.yaml or the location in the zopeneditor.zowe.defaultRseConversionMappingsFile setting).
2. If no RSE mappings exist or if you are using a z/OSMF profile to access the file, Z Open Editor will check for any matches between the file's path and the patterns defined in the zopeneditor.encodings.filePatterns setting.
   • This setting maps filename glob patterns to an EBCDIC file encoding. These patterns should be specified as glob patterns. A single asterisk * matches any number of characters within a path segment, including none. A double asterisk ** matches any number of path segments, including none. For example, { **/COBOL/*.cbl : IBM-930 } will assign any file paths containing a directory named COBOL ending with a .cbl extension an encoding of IBM-930. Learn more about how to specify glob patterns.
3. If the file path does not match of the patters in the setting, the encoding specified in the Zowe profile is used.
4. Local files will only be matched against the zopeneditor.encodings.filePatterns setting for an encoding.

These settings only control what encodings are passed to the language servers. To control encodings used for file conversions when uploading or downloading z/OS resources, use your ZAPP file or Zowe profile.