How can I redirect input in PowerShell without a BOM?

Question:

I am trying to redirect input in PowerShell by:

The problem is the piped UTF-8 text is preceded with a BOM (0xEFBBBF), and my program cannot handle that correctly.

A minimal working example:

Then in PowerShell:

Or simply

In either case, the output is EF BB BF 41 42 43 0D 0A.

How can I pipe the text into the program without 0xEFBBBF?

Answer:

Note:
The following contains general information that in a normally functioning PowerShell environment would explain the OP’s symptom. That the solution doesn’t work in the OP’s case is owed to machine-specific causes that are unknown at this point.
This answer is about sending BOM-less UTF-8 to an external program; if you’re looking to make your PowerShell console windows use UTF-8 in all respects, see this answer.

To ensure that your Java program receives its input UTF-8-encoded without a BOM, you must set $OutputEncoding to a System.Text.UTF8Encoding instance that does not emit a BOM:

Caveats:

  • Do NOT use the seemingly equivalent New-Object Text.Utf8Encoding $false, because, due to the bug described in this GitHub issue, it won’t work if you assign to $OutpuEncoding in a non-global scope, such as in a script. In PowerShell v4 and below, use
    (New-Object Text.Utf8Encoding $false).psobject.BaseObject as a workaround.
  • Windows 10 version 1903 and up allow you to set BOM-less UTF-8 as the system-wide default encoding (although note that the feature is still classified as beta as of version 20H2) – see this answer; [fixed in PowerShell 7.1] in PowerShell [Core] up to v7.0, with this feature turned on, the above technique is not effective, due to a presumptive .NET Core bug that causes a UTF-8 BOM always to be emitted, irrespective of what encoding you set $OutputEncoding to (the bug is possibly connected to this GitHub issue); the only solution is to turn the feature off, as shown in imgx64’s answer.

If, by contrast, you use [Text.Encoding]::Utf8, you’ll get a System.Text.Encoding.UTF8 instance with BOM – which is what I suspect happened in your case.


Note that this problem is unrelated to the source encoding of any file read by Get-Content, because what is sent through the PowerShell pipeline is never a stream of raw bytes, but .NET objects, which in the case of Get-Content means that .NET strings are sent (System.String, internally a sequence of UTF-16 code units).

Because you’re piping to an external program (a Java application, in your case), PowerShell character-encodes the (stringified-on-demand) objects sent to it based on preference variable $OutputEncoding, and the resulting encoding is what the external program receives.

Perhaps surprisingly, even though BOMs are typically only used in files, PowerShell respects the BOM setting of the encoding assigned to $OutputEncoding also in the pipeline, prepending it to the first line sent (only).

See the bottom section of this answer for more information about how PowerShell handles pipeline input for and output from external programs, including how it is [Console]::OutputEncoding that matters when PowerShell interprets data received from external programs.


To illustrate the difference using your sample program (note how using a PowerShell string literal as input is sufficient; no need to read from a file):

In Windows PowerShell, where $OutputEncoding defaults to ASCII(!), you’d see the following with the default in place:

Note that 3F represents the literal ? character, which is what the non-ASCII ö character was transliterated too, given that it has no representation in ASCII; in other words: information was lost.

PowerShell [Core] v6+ now sensibly defaults to BOM-less UTF-8, so the default behavior there is as expected.
While BOM-less UTF-8 is PowerShell [Core]’s consistent default, also for cmdlets that read from and write to files, on Windows [Console]::OutputEncoding still reflects the active OEM code page by default as of v7.0, so to correctly capture output from UTF-8-emitting external programs, it must be set to [Text.UTF8Encoding]::new($false) as well – see this GitHub issue.

Source:

How can I redirect input in PowerShell without a BOM? by licensed under CC BY-SA | With most appropriate answer!

Leave a Reply