I am looking at this:
function encodeCodePoint(codePoint) {
if ((codePoint & 0xFFFFFF80) == 0) { // 1-byte sequence
return stringFromCharCode(codePoint);
}
var symbol = '';
if ((codePoint & 0xFFFFF800) == 0) { // 2-byte sequence
symbol = stringFromCharCode(((codePoint >> 6) & 0x1F) | 0xC0);
}
else if ((codePoint & 0xFFFF0000) == 0) { // 3-byte sequence
checkScalarValue(codePoint);
symbol = stringFromCharCode(((codePoint >> 12) & 0x0F) | 0xE0);
symbol += createByte(codePoint, 6);
}
else if ((codePoint & 0xFFE00000) == 0) { // 4-byte sequence
symbol = stringFromCharCode(((codePoint >> 18) & 0x07) | 0xF0);
symbol += createByte(codePoint, 12);
symbol += createByte(codePoint, 6);
}
symbol += stringFromCharCode((codePoint & 0x3F) | 0x80);
return symbol;
}
which, in JavaScript, seems to be taking advantage of the fact that the numbers in JavaScript are (I think) something around 32-bits long. So it does some bit manipulation which I am unfamiliar with and gets the encoded value. Same with the decode
function:
function decodeSymbol() {
var byte1;
var byte2;
var byte3;
var byte4;
var codePoint;
if (byteIndex > byteCount) {
throw Error('Invalid byte index');
}
if (byteIndex == byteCount) {
return false;
}
// Read first byte
byte1 = byteArray[byteIndex] & 0xFF;
byteIndex++;
// 1-byte sequence (no continuation bytes)
if ((byte1 & 0x80) == 0) {
return byte1;
}
// 2-byte sequence
if ((byte1 & 0xE0) == 0xC0) {
byte2 = readContinuationByte();
codePoint = ((byte1 & 0x1F) << 6) | byte2;
if (codePoint >= 0x80) {
return codePoint;
} else {
throw Error('Invalid continuation byte');
}
}
// 3-byte sequence (may include unpaired surrogates)
if ((byte1 & 0xF0) == 0xE0) {
byte2 = readContinuationByte();
byte3 = readContinuationByte();
codePoint = ((byte1 & 0x0F) << 12) | (byte2 << 6) | byte3;
if (codePoint >= 0x0800) {
checkScalarValue(codePoint);
return codePoint;
} else {
throw Error('Invalid continuation byte');
}
}
// 4-byte sequence
if ((byte1 & 0xF8) == 0xF0) {
byte2 = readContinuationByte();
byte3 = readContinuationByte();
byte4 = readContinuationByte();
codePoint = ((byte1 & 0x07) << 0x12) | (byte2 << 0x0C) |
(byte3 << 0x06) | byte4;
if (codePoint >= 0x010000 && codePoint <= 0x10FFFF) {
return codePoint;
}
}
throw Error('Invalid UTF-8 detected');
}
Basically, I can't quite read this code and can't really tell what's going on. Wondering if one with better bit-manipulation chops or UTF-8 encoding knowledge could describe at a high level what the input and output are from encoding and decoding, and very roughly how it goes from input to output for each. I am trying to build a utf-8 encoder/decoder and don't see exactly how an 8-bit stream is chunked into 1 to 4 byte chunks, partly because the JavaScript 32-bit integer thingy is getting in the way I think. But to me it seems like this is what happens:
Decoding:
String.fromCharCode(integer)
sort of thing.What I'm missing is how exactly it goes from the 1-byte sequence to up to 4 bytes, how does it do that part?
Encoding:
Wondering if one could fill in the gaps in my understanding. I'm not looking for exactly each bit-manipulation step, as there are a lot. I am just looking for the questions which I highlighted in my analysis just above.
JS integers have 32bit binary operators, thus you can safely work with 4 x 8bit (4bytes) in one single number. That's what your decoder receives as a parameter.
UTF-8 encoding is variable in size. If the codepoint would only take 7bits (= ASCII), then it would fit into one byte, that has a leading zero to indicate that it only has one byte:
0XXXXXXXX
Now to check whether the codepoint is only one byte, one could check if there is a bit set somewhere in the upper bytes. That can be done by comparing the codepoint to 0xFFFFF80
, which has all bits set excluding the last 8. Thus, if a bitwise and results in something unequal 0, there is a bit set somewhere in the upper bytes.
1111111111111111111110000000 &
0XXXXXXX
= 0
Now if there are more than 7 bits, the first byte contains the number of bytes, all the following bytes contain a 01
sequence at the beginning, for 4 bytes that would be:
11110XXX 10XXXXXX 10XXXXXX 10XXXXXX
Now to get the upper 8 encoded bits here for example, one could rightshift by 18:
1110XXX 10XXXXX
User contributions licensed under CC BY-SA 3.0