Dev Diary RSS Feed

Exclude directories (and avoid traversing) with find

Tue, 06 Jan 2026 11:00:00 GMT

$ find . \( -name "node_modules" \
    -or -name ".git" \) -prune \
  -or -type f -print

And this for directories:

$ find . \( -name "node_modules" \
    -or -name ".git" \) -prune \
  -or -type d -print

So, if you want to ignore directories, a good rule of thumb is to put the prune stuff first, and then specify what you want to print. Additionally, don’t forget to put an explicit print at the end. See the questions below for more.

If you have many paths you can use regex:

$ find . \
    -regextype posix-extended \
    -type d \
    -regex ".*\/\.?(node_modules|git|public|cache|next|coverage).*" -prune \
  -or \
    -type d -print

$ find . \
    -regextype posix-extended \
    -type d \
    -regex ".*\/\.?(node_modules|git|public|cache|next|coverage).*" -prune \
  -or \
    -type f -print

Note that you can put more filters after -type and before the ending -print.

Some things to keep in mind

The find expressions are evaluated from left to right: “GNU find searches the directory tree rooted at each given starting-point by evaluating the given expression from left to right”
If we have 2 expressions separated by -or and the 1st one is true, find won’t evaluate the 2nd expression.
If you don’t specify a command, find appends a -print at the end.
-prune all it does is to skip traversing the directory tree.
An expression that contains -prune at the end, returns true if it prunes and false if it doesn’t prune.

Question 1: Why `find . $ -name "node_modules" -or -name ".git" $ -prune` still prints the `node_modules` and `.git` top-level directories?

$ find . \( -name "node_modules" \
  -or -name ".git" \) -prune

The man page of find states about -prune that “if the file is a directory, do not descend into it”, that’s what it does, nothing more. Also, If the only action is -prune, find still appends a -print that will print the pruned directory name (and avoid descending further into it off course, as a side-effect).

So, if you don’t want to print the pruned top-level directory, append at the end -or expression -print where expression is what you want to print. This way find won’t automatically append a print at the end.

Question 2: Why `find . -type f -print -o $ -name node_modules -o -name .git $ -prune` gives the same output as the “canonical form” `find . $ -name node_modules -o -name .git $ -prune -o -type f -print`?

$ find . -type f -print -o \
  \( -name node_modules -o -name .git \) -prune

$ find . \( -name node_modules -o -name .git \) -prune \
  -o -type f -print

The above commands give the same result and it might seem strange at first.

The 2nd expression, aka the “canonical form” (find . $ -name node_modules -o -name .git $ -prune -o -type f -print), behaves the way it does, because, if the item is a regular file it will skip the 1st part, it will go to the 2nd and will print it (for the case when it’s a file and it’s inside a prune directory, keep reading, I explain it further down). If the item is a directory and it’s on the prune directories, it will stop descending further. If it’s a directory and it’s not on the prune directories, it will do nothing.

Now, I want to focus on the 1st expression, and explain how it works.

This question can also take the following form: Why the 1st part of the 1st expression, -type f -print -o doesn’t first print all the files, even those inside the pruned directories? The reason is probably because the find expression is evaluated on each file, and the files are sorted lexicographically ascending by name. Say, for example, you have an index.js file, that starts from i, and a node_modules folder, that starts from n, with ton of sub-folders and files. One of those enclosed files might be a node_modules/some-package/index.js file, which is after node_modules in order. i is before n in the sorting order, so the index.js file will be evaluated first. It’s file, so , because of the 1st part, it’s printed. We then proceed with the top level folder node_modules, that’s not a file, but a directory, so we don’t print it. It matches the 2nd part of the find expression, and it also has a prune command, so we won’t descend into it’s subdirectories and files. It doesn’t have a print command, we won’t print node_modules.

So that’s why we won’t print node_modules/some-package/index.js, we first run the expression on the parent folder node_modules that’s pruned and we don’t descend further down.

Question 3: Why `find . -type d -print -o $ -name node_modules -o -name .git $ -prune` output is not the same as `find . $ -name node_modules -o -name .git $ -prune -o -type d -print`? The 2 corresponding commands were equivalent for files (-type f).

$ find . -type d -print -o \
  \( -name node_modules -o -name .git \) -prune

$ find . \( -name node_modules -o -name .git \) -prune \
  -o -type d -print

Let’s start by stating that both of the above commands won’t print any files, they will print only directories, so we don’t have to worry about that.

2nd expression

In the 2nd expression, the “canonical form” (find . $ -name node_modules -o -name .git $ -prune -o -type d -print), in the 1st part of the expression, if the directory is in the prune list, find won’t further traverse it. Also, -prune returns true in this case, so we won’t evaluate the 2nd part of the expression (short-circuits due to -or). We didn’t have to make a note about the fact that -prune returns true in the corresponding case in Q2, about the files, because it was self-evident. We were printing files, not directories, so even if -prune returned false (which is not the case) and the 2nd part of the expression was evaluated, we would not have printed the directory anyway.

If the directory is not in the prune list, the 1st part of the expression (the -prune) returns false, and, because we have an -or, we also evaluate the 2nd part of the expression, and we print the directory.

1st expression

In the 1st expression (find . -type d -print -o $ -name node_modules -o -name .git $ -prune), if the directory is in the prune list, the 1st part will print it (incorrectly), it will return true and it won’t evaluate the 2nd part.

If the directory is not in the prune list, again, the 1st part of the expression will print it (correctly this time), it will return true and it won’t evaluate the 2nd part.

So, the 1st expression just prints all the directories and doesn’t prune anything. You can test that it prints all the directories with the following command: diff <(find . -type d -print -o $ -name node_modules -o -name .git $ -prune) <(find . -type d), the diff will be empty, so the 2 commands are the same.

Question 4: Why use `-prune` and not `-not -path "pathname"`

The man page of find suggests to use -prune if you want to exclude directories. If you use something like -not -path "pathname", you will exclude the directory, but you will still traverse it, and, as a result, the search will be slow for big directories like node_modules.

Array decay to pointers, and address of operator (dereference) behavior.

Mon, 05 Jan 2026 14:00:00 GMT

Address of operator

The address of operator (&) creates a pointer of the underlying type. For example:

int number = 5;
auto what = &number;
// "what" is an `int *`, a pointer to the underlying int

auto what = number
// "what" is an `int`

Array decay to pointer

int numbers[10] = { 0 };
auto what = numbers
// "what" is not an int[10] because array decay to pointers
// when evaluated in an expression. So "what" decays
// from `int[10]` to `int *`
auto what = &numbers;
// "what" is `int (*)[10]`, a pointer to an array of 10
// ints. Here, the address of operator prevents the decay
// and creates a pointer to the underlying type, which is
// an array of 10 ints.
auto what = numbers[1];
// "what" is an int

Multi-dimensional array

int matrix[2][3] = { {1, 2, 3}, {4, 5, 6} };
auto what = matrix;
// matrix is an array that holds 2 arrays that, in turn,
// hold 3 ints. When you have an array, of any type, in an
// expression, that array decays to a pointer.
// So, "what" is now `int (*) [3]` a pointer to an array of 3 ints.
// Goodbye to the array of 2, now it's a pointer.

auto what = &matrix;
// This time, the address-of operator prevents the decay
// and "what" is a pointer to the original type, which
// is int[2][3], or an array that holds 2 arrays that hold 3 ints.
// So, technically, "what" is an `int (*) [2][3]`.

auto what = matrix[0];
// matrix [0] is an int[3] array and we know that array
// decay to pointers in expressions.
// So, "what" is an `int *` here.

auto what = &matrix[0];
// Again, matrix[0] is of type `int[3]`. The address-of
// operator prevents the decay of the `int[3]` to `int *`
// and, in addition,  it gives us a pointer to the `int [3]`.
// So, "what" is of type `int (*) [3]`, Or a pointer to an
// array of 3 ints.

auto what = matrix[0][0];
// "what" is `int` here.

auto what = &matrix[0][0];
// `matrix[0][0]` is an int, so `&matrix[0][0]` is a
// pointer to int (`int *`).

decltype

Btw, in C++, expressions in decltype don’t decay the same way arrays decay to pointers in expressions

int matrix[2][3] = { {1, 2, 3}, {4, 5, 6} };
using what = decltype(matrix);
// "what" is a reference to an `int matrix[2][3]`, or a
// `int (&) [2][3]`.

using what = decltype(matrix[0]);
// what is a `int (&) [3]`

using what = decltype(matrix[0][0]);
// what is a `int &`

int number = 5;
using what = decltype(number);
// what is `int`. Don't ask me why it's different from the
// above.

Something to run to observe the above:

int main()
{
    int matrix[2][3] = { { 1, 2, 3 }, { 4, 5, 6 } };
    int* pi = matrix[0];
    int** ppi = &pi;
    int number = 7;
    std::cout << "           pi: " << pi << '\n';
    std::cout << "          *pi: " << *pi << '\n';
    std::cout << "          &pi: " << &pi << '\n';
    std::cout << "       pi + 1: " << pi + 1 << '\n';
    std::cout << "    *(pi + 1): " << *(pi + 1) << '\n';

    std::cout << "          ppi: " << ppi << '\n';
    std::cout << "         *ppi: " << *ppi << '\n';
    std::cout << "         &ppi: " << &ppi << '\n';
    std::cout << "      ppi + 1: " << ppi + 1 << '\n';
    std::cout << "   *(ppi + 1): " << *(ppi + 1) << '\n';

    std::cout << "       number: " << number << '\n';
    std::cout << "      &number: " << &number << '\n';

    std::cout << "       matrix: " << matrix << '\n';
    std::cout << "      &matrix: " << &matrix << '\n';
    std::cout << "   &matrix[0]: " << &matrix[0] << '\n';
    std::cout << "&matrix[0][0]: " << &matrix[0][0] << '\n';
    std::cout << "&matrix[0][1]: " << &matrix[0][1] << '\n';
    std::cout << "&matrix[0][2]: " << &matrix[0][2] << '\n';
    std::cout << "   &matrix[1]: " << &matrix[1] << '\n';
    std::cout << "&matrix[1][0]: " << &matrix[1][0] << '\n';
    std::cout << "&matrix[1][1]: " << &matrix[1][1] << '\n';
    std::cout << "&matrix[1][2]: " << &matrix[1][2] << '\n';
    return 0;
}

The naming of the what variable was probably poor, I had to enclose it in parentheses to make it clear I was referring to the variable. I always thought of “So What” by Anti-Nowhere League/Metallica.

C++ Why we add parenthesis around assignments in if statements

Mon, 05 Jan 2026 13:00:00 GMT

I made a note for the question in the title back in 2022, but I always forget the answer.

From C++ Primer, 5th edition 2013, 4.4 “Assignment Operators” (p.146):

Because assignment (=) has lower precedence than the relational operators (<><=, etc.), parentheses are usually needed around assignments in conditions.

int i;
while ((i = get_value()) != 42) {
  // do something . . .
}

You can verify it with man pages:

$ apropos associativity
associativity: nothing appropriate.
$ apropos preced
operator (7)         - C operator precedence and order of evaluation
precedence (7)       - C operator precedence and order of evaluation
$ man 7 operator
...
NAME
       operator - C operator precedence and order of evaluation

DESCRIPTION
       This manual page lists C operators and their precedence in evaluation.

       Operator                            Associativity   Notes
       [] () . -> ++ --                    left to right   [1]
       ++ -- & * + - ~ ! sizeof            right to left   [2]
       (type)                              right to left
       * / %                               left to right
       + -                                 left to right
       << >>                               left to right
       < > <= >=                           left to right
       == !=                               left to right
       &                                   left to right
       ^                                   left to right
       |                                   left to right
       &&                                  left to right
       ||                                  left to right
       ?:                                  right to left
       = *= /= %= += -= <<= >>= &= ^= |=   right to left
       ,                                   left to right

Also an extra bit from C++ Primer p.145

The result of an assignment is its left-hand operand, which is an lvalue. The type of the result is the type of the left-hand operand. If the types of the left and right operands differ, the right-hand operand is converted to the type of the left.

Expressions become statements if you add a semicolon at the end of them (K&R C book)

Mon, 05 Jan 2026 12:00:00 GMT

Two things I found interesting and wrote down while reading the K&R C book (The C Programming Language by Ritchie & Kernighan), back in May 2024. I revisited again the note in 2026, and to be honest, I had forgot the 1st.

1. expressions vs statements

Expressions become statements if you add a semicolon at the end of them, K&R C book, Chapter 3.1, “Statements and Blocks”:

An expression such as x = 0 or i++ or printf(...) becomes a statement when it is followed by a semicolon, as in:

 x = 0;
 i++;
 printf(...);

In C, the semicolon is a statement terminator, rather than a separator as it is in languages like Pascal.

For more types of statements in C++ (we saw expression statements here), see https://en.cppreference.com/w/cpp/language/statements

2. braces

Also braces {} combine multiple statements into compound statements, which is another statement type.

Braces { and } are used to group declarations and statements together into a compound statement, or block, so that they are syntactically equivalent to a single statement. The braces that surround the statements of a function are one obvious example; braces around multiple statements after an if, else, while, or for are another. (Variables can be declared inside any block; we will talk about this in Chapter 4.) There is no semicolon after the right brace that ends a block.

Work with binary option flags (flipping bits)

Sat, 03 Jan 2026 12:00:00 GMT

| = bitwise or
& = bitwise and
~ = not, negation
<=> = “if and only if”, aka left and right are both true under same conditions.
set or setting a flag means we want the flag to be 1.
unset/clear or clearing a flag means we want it to be 0.

1) Set options

Set all 3 options: OPTION_1 | OPTION_2 | OPTION_3.

For example: 001 | 010 | 100 = 111, so all 3 options are set.

Or, set a new option and add it to the current options: CURRENT_OPTIONS | NEW_OPTION.

For example, if the 1st/2nd bit options are already set and we want to also set the 3rd bit option: 011 | 100 = 111.

2) Unset options

CURRENT_OPTIONS & ~OPTION_1.

For example: 101 & ~001 <=> 101 & 100 = 100.

We get he same result even if OPTION_1 is not already set. For example, 100 & ~001 <=> 100 & 100 = 100.

3) Check if an option is set

See if OPTION_2 is set: CURRENT_OPTIONS & OPTION_2.

For example: 101 & 010 = 000 (falsey, not set), or 110 & 010 = 010 (OPTION_2, truthy and set).

The above means that you can use it inside an if control flow: if (CURRENT_OPTIONS & OPTION_1) {}. If it’s not set, it will be 0, if it’s not set it will be truthy and equal to OPTION_1.

vcpkg list dependencies from the manifest (vcpkg.json), not global dependencies.

Sat, 03 Jan 2026 11:00:00 GMT

I installed vcpkg at /c/src/vcpkg/vcpkg.exe, change the command accordingly.

$ /c/src/vcpkg/vcpkg.exe list \
    --x-wait-for-lock \
    --triplet "x64-windows" \
    --vcpkg-root=/c/src/vcpkg \
    --x-manifest-root='.\MyProject\' \
    --x-install-root='.\MyProject\vcpkg_installed\x64-windows\'

or save yourself some time from typing by going first to the folder where vcpkg.json lives:

$ cd .\MyProject\
$ /c/src/vcpkg/vcpkg.exe list \
    --x-wait-for-lock \
    --triplet "x64-windows" \
    --x-install-root='.\vcpkg_installed\x64-windows\'

Edit binary files with dd: replacing, deleting, and copying bytes

Thu, 01 Jan 2026 12:00:00 GMT

Replace bytes with dd (hex editor, edit bytes)

# The file you want to edit:
$ echo 'My name is dark' > file.txt
$ cat file.txt
My name is dark

# Replace 1 byte:
$ echo -n 'M' |
  dd \
    of=file.txt  \
    bs=1 \
    seek=11 \
    count=1 \
    conv=notrunc

1+0 records in
1+0 records out
1 byte copied, 5.1489e-05 s, 19.4 kB/s

# Inpect file:
$ cat file.txt
My name is Mark
$ xxd file.txt
00000000: 4d79 206e 616d 6520 6973 204d 6172 6b0a  My name is Mark.

Notes:

stdin = input file = ‘if’, in other words, what you want to add to the new file
of is the file you want to edit.
bs, read and write up to BYTES bytes at a time (default: 512); overrides ibs and obs
seek, The decimal address you want to start replacing. If you want to add a hex address, wrap in double quotes and use arithmetic substitution$((0x30)).
count, how many bytes from the ‘if’ you want to add to the ‘of’.
conv=notrunc, do not truncate the file, which is the default.

Delete bytes with dd

It seems that you can’t remove individual bytes. If you think about it, there isn’t a system call that does this.

What you can do though is to write the whole file again, without those bytes:

 $ echo 'helllo doge world ' |
  { dd bs=1 count=4 ; dd skip=1 bs=1 count=2 ; dd skip=5 bs=1 ; }  2>/dev/null

Explanation:

1. print hell
1. skip 3rd l, print ‘o’ and space.
1. delete doge and space and print world.

See https://unix.stackexchange.com/questions/128871/dd-remove-range-of-bytes

Copy a part of a file with dd

$ dd if=input.txt iflag=skip_bytes,count_bytes skip=5 count=3 of=part.txt
# or
$ dd if=input.txt bs=1 skip=5 count=3 of=part.txt

There are 2 versions because if you use the skip/count combo without a block size (bs) skip=N and count=N, the N is a block and not bytes. So, if you want to skip bytes, and you don’t want to change the block size, use the 1st option (set the block size to 1 byte). Otherwise, you can use the second option.

Download git submodules

Sun, 28 Dec 2025 14:00:00 GMT

To see the git submodules of the project:

$ git submodule status

Add a submodule from git:

$ git submodule add git@github.com:oatpp/oatpp.git modules/oatpp

If you already have cloned the project:

$ git submodule update --init

You can also break it up in two steps:

$ git submodule init
$ git submodule update

or to clone and download submodules:

$ git clone --recursive https://github.com/obsproject/obs-studio.git

For more see: https://www.git-scm.com/book/en/v2/Git-Tools-Submodules

Git rebase and create new timestamps (dates)

Sun, 28 Dec 2025 13:00:00 GMT

$ git checkout development
$ git rebase --ignore-date origin/development

what --ignore-date does:

Instead of using the author date of the original commit, use the current time as the author date of the rebased commit.

The opposite is the --committer-date-is-author-date option, which is the default, in my system:

Instead of using the current time as the committer date, use the author date of the commit being rebased as the committer date.

Parsing JSON/XML/YML in the command line with jq/yq/xq

Sun, 28 Dec 2025 12:00:00 GMT

Install

Install jq with a package manager. For example, in Ubuntu with apt:

$ sudo apt install jq

The xq/yq executables are part the python package yq https://github.com/kislyuk/yq that transforms the XML/YML files to JSON and calls jq with the filter you provide. Install it with

# You need to install python3 first that includes the pip package manager.
$ pip3 install yq

xq/yq call jq, so the filters will be the same.

See https://stedolan.github.io/jq/manual/

Get the first item of the array

$ curl https://jsonplaceholder.typicode.com/users | jq .[0]

Get only a specific field from the array items

Also count the unique

$ curl https://jsonplaceholder.typicode.com/users |
 jq .[].address.city | sort | uniq -c

Get the length

Get the length of an array.

$ cat facebook.xml | xq  '.rss.channel.item | length'

Get the length of nested fields. Be careful, though, because if the field is a string, instead of an array, you’ll get the string length.

$ cat facebook.xml | xq  '.rss.channel.item[]."g:additional_image_link" | length'

Filter entries with select

$ curl https://jsonplaceholder.typicode.com/users |
 jq '.[] | select(.username == "Bret")'

Use and/or:

$ curl https://jsonplaceholder.typicode.com/users |
 jq '.[] | select(.username == "Bret" and .address.city == "Gwenborough")'

Unescape character from JSON string with fromjson filter (parse)

$ cat | jq '. | fromjson'

> "{\"result\":[{\"battery.current\":0,\"vehicle.mileage\":437.974}]}"

prints:

{
 "result": [
   {
     "battery.current": 0,
     "vehicle.mileage": 437.974
   }
 ]
}

Or decode JSON strings in your programming language.

Return specific fields in a row with string interpolation

$ curl https://jsonplaceholder.typicode.com/users |
 jq '.[] | select(.username == "Bret") | "\(.id) \(.name) \(.company.name)"'

Pick specific fields from an object

$ jq '.urlset.url[] | { loc, lastmod }' kids-shop-123-sitemap.json | head

{
 "loc": "https://kids-shop-123.gr/kathisma-daxtilidi-baniou-panda-prasino-lorelli",
 "lastmod": "2023-04-27T21:54:05+00:00"
}
{
 "loc": "https://kids-shop-123.gr/kathisma-daxtilidi-baniou-panda-roz-lorelli",
 "lastmod": "2023-02-28T13:52:06+00:00"
}
{
 "loc": "https://kids-shop-123.gr/1022057-Lorelli-pipila-silikonis-MY-FRIEND-me-kallyma1022057-blue",

Select with pipes and match

Note the pipe inside select(..).

$ cat facebook.xml |
 xq  '.rss.channel.item[] | select(."g:link" | match("https:\/\/kids-shop-123.gr\/.*\/"))."g:link"' |
 grep -ivE route

Select with pipes and contains (includes)

$ cat facebook.xml |
 xq  '.rss.channel.item[] | select(."g:link" | contains("Lorelli")) | ."g:link"'

Get all the possible properties (keys)

From an array of objects with keys built-in function.

$ cat facebook.xml | xq  '.rss.channel.item[] | keys' | sort | uniq

You can also use keys[], I don’t know atm what’s the difference.

Get all the possible values (values)

From an array of objects with values[] built-in function.

$ jq '.[][1].ips | values[]' ./x64/Debug/user-stats.json
# From input:
[
 [
   667204,
   {
     "ips": {
       "122.127.33.1": 1685028929
     }
   }
 ],
 [
   675978,
   {
     "ips": {
       "10.26.218.58": 1685028929
     }
   }
 ],
...
# Output:
1685028929
1685028929
1685028976
...

Get the items that their value is an array

Use the type

$ cat shop-2023-06-17.json |
 jq '.products.product[] | select( .manufacturer == "CRISTIANA MASI") | select( values[] | type == "array" )'

Find which items have a value that is an array, and get the items’ keys and counts

to_entries transforms an object to an array of key/value objects. It’s usefull if you want to do stuff with one of the key/value and then reference the other.

$ cat ~/Desktop/shop/shop-2024-02-10.json |
 jq -r '.data.post[] | to_entries | .[] | select(.value | type == "array") | "\(.key) \(.value | length)"' |
 sort |
 uniq -c |
 sort |
 awk 'BEGIN{print "count, key, array_length"} {print $0}'

# prints
count, key, array_length
     1 ImageFilename 3
     1 ImageFilename 8
     1 ImageURL 3
     1 ImageURL 8
     1 ΠροϊόνΣύνθεση 2
     2 ΠροϊόνΔιαστάσεις 2
     2 ΠροϊόνΧρώμα 4
     3 Κατηγορίεςπροϊόντων 3
    12 ΠροϊόνΧρώμα 2
    21 ImageFilename 2
    21 ImageURL 2
    55 Κατηγορίεςπροϊόντων 2

The use case here is to see which objects from an array have array properties. At the same time, print the key and the length of the array. Useful if I want to use wpallimport for a feed to see how many max additional images or attributes to account for (and not lose existing data).

$ cat kids-shop-123-facebook-2023-06-19.json |
 jq -c '.rss.channel.item[] | to_entries[] | if .value | type == "array" then { key, length: .value | length } else null end' |
 grep -v null | head

Drop (exclude, delete, remove) a property (field) from an object path

Use the del(path_expression).

$ echo '[
 { "name": "Mark", "age": 12, "password": "123456" },
 { "name": "John", "age": 21, "password": "123" }
]' |
 jq 'del(.[].password)'

# prints
[
 {
   "name": "Mark",
   "age": 12
 },
 {
   "name": "John",
   "age": 21
 }
]

Add a property (field) to an object

Add the profit_target_sent property to all objects, with a default value of 0.

$ jq '.records[][1] += { "profit_target_sent": 0.0 }' my_file.json > my_file.json.ready

Sort results

Notice that it’s item, not item[].

$ cat facebook.xml |
 xq  '.rss.channel.item | sort_by(."g:quantity") | .[]."g:quantity"'

Sort results and keep the initial structure

Use |= vs |: See https://stackoverflow.com/a/30332672

$ jq '.urlset.url |= sort_by (.loc)' kids-shop-123-sitemap-real.json | head

{
 "urlset": {
   "@xmlns": "http://www.sitemaps.org/schemas/sitemap/0.9",
   "@xmlns:image": "http://www.google.com/schemas/sitemap-image/1.1",
   "url": [
     {
       "loc": "https://kids-shop-123.gr/10010060003-marsipo-lorelli-prasino",
       "changefreq": "weekly",
       "lastmod": "2021-11-03T13:49:11+00:00",
       "priority": "1.0",

Explanation:

$ echo '{ "numbers": [{ "value": 2 }, { "value": 3 }, { "value": 1 }] }' |
 jq '.numbers | sort_by ( .value )'
# | (pipe) did not keep numbers property.
[
 {
   "value": 1
 },
 {
   "value": 2
 },
 {
   "value": 3
 }
]

$ echo '{ "numbers": [{ "value": 2 }, { "value": 3 }, { "value": 1 }] }' |
 jq '.numbers |= sort_by ( .value )'

# |= kept numbers property.
{
 "numbers": [
   {
     "value": 1
   },
   {
     "value": 2
   },
   {
     "value": 3
   }
 ]
}

How to remove items from a nested array based on a filter and keep the rest structure unchanged

Again, use the assignment operator (|=)

$ echo '{"message":"Hi","items":[1,2,3]}' |
 jq -c '.items[] |= select(. >2)'

{"message":"Hi","items":[3]}

Note this works for jq version >= 1.7

Update with pipe assignment operator |= multiple properties at once, maybe even on different levels of nesting.

You do what you did above, but you then use regular pipe to update the second property.

$ echo '{"message":"Hi","items":[1,2,3],"hello":{"items":[1,4,5]}}' |
 jq -c '.items[] |= select(. >2) | .hello.items[] |= select(. >2)'

{"message":"Hi","items":[3],"hello":{"items":[4,5]}}

Unique field

It’s not a filter, the same as sort_by

cat facebook.xml |
 xq  '.rss.channel.item | unique_by(."g:quantity") | .[]."g:quantity"'

How to minify a JSON file

Use the -c (compact) option:

$ echo '{
  "message": "Hello World",
  "number": 123
}' | jq -c '.'

# prints:
{"message":"Hello World","number":123}

Count objects if the output is an object stream (?) instead of an array with –compact, -c option (keywords: minify, minified, minimize, compress)

The length works if the output is an array:

$ cat input/current-shop-export-22-09-2022.xml | xq '.data.post | length'
# Prints 761.

But if the output consists of individual objects, the length is applied as a filter on each object:

$ cat input/current-shop-export-22-09-2022.xml | xq '.data.post[] | length'
# Prints:
# 25
# 25
# ...

This comes up when you already filtered the array and you want to count how many items satisfy the filter. In this case, you can use the -c (compact) option to print each item in a single line and pipe the output to wc to count the number of lines

$ cat input/current-shop-export-22-09-2022.xml |
 xq -c  '.data.post[] | select(."Ετικέτεςπροϊόντος" | type == "array")' |
 wc -l
# Prints 208

Convert JSON to csv with jq

See https://stackoverflow.com/questions/32960857/how-to-convert-arbitrary-simple-json-to-csv-using-jq

$ jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv' input.json > output.csv

Get the results as raw strings (without quotes)

Say you don’t want the string quotes:

$ echo '["A", "B", "C"]' | jq '.[]'

"A"
"B"
"C"

Use the -r option:

$ echo '["A", "B", "C"]' | jq -r '.[]'

A
B
C

-r               output raw strings, not JSON texts;

Multiple expresssions (search keyword: conditions) in select

I want to select the tags from Woocommerce products that are arrays (they can also be a single string) and their length is equal to 8:

$ cat input/current-shop-export-22-09-2022.xml |
 xq  '.data.post[] | select(."Ετικέτεςπροϊόντος" | (type == "array") and length == 8)'

Enclose the first expression in parenthesis and use the and keyword for the second expression, see https://stackoverflow.com/questions/33057420/jq-select-multiple-conditions, https://stedolan.github.io/jq/manual/#select(boolean_expression)

jq ternary if then else elseif conditions

My product categories are either a single category (string) or an array of categories (strings). I want to return the single category string, but I want to flatten the category arrays. For that reason, I will use if-then-else, see https://stedolan.github.io/jq/manual/#if-then-else and the example below:

$ cat input/current-shop-export-22-09-2022.xml |
 xq -r '.data.post[]."Κατηγορίεςπροϊόντων" | if type == "array" then .[] else . end' |
 sort | uniq -c

SQL-like operators, IN: check if the value of a property is in an array of known values

$ jq '.records[] | select(.[0] | IN(20008,20009,20010))' profits.json

Get a range from an array

Use the [start_index:end_endex] syntax:

# get the first 2 million elements from the array:

$ jq '.[0:2000000]' results-1705071239.json > results-1705071239-first-2m.json

You can also query items from the end of the array by adding negative indices.

You can also omit the end index if you from the first element until the last element of the array.

See “jq: Select range” https://stackoverflow.com/questions/45548604/jq-select-range

Combine arrays with jq with the slurp option

# inputs is
# [] length 200
# [] length 50
# [] length 550
# that resulted from multiple files

$ jq 'transform-stuff' *json | jq -s 'add' input.json

# prints
[each object merge here] length 800

-s option => -s read (slurp) all inputs into an array; apply filter to it;

The filter in this case is add.

More examples to understand what’s happening:

# this works but our data is not in this form
$ echo '[[1,2],[3,4]]' | jq 'add'
[
 1,
 2,
 3,
 4
]

# our data is in this form and the output is not what we want...
# it applies the filter on each individual array.
$ echo '[1,2][3,4]' | jq 'add'
3
7

# jq slurp to the rescue:
$ echo '[1,2][3,4]' | jq -s 'add'
[
 1,
 2,
 3,
 4
]

# just jq -s also doesn't work in case you're wondering
$ echo '[1,2][3,4]' | jq -s
[
 [
   1,
   2
 ],
 [
   3,
   4
 ]
]

Links

See https://stedolan.github.io/jq/manual/ and https://remysharp.com/drafts/jq-recipes

C++ constructor/destructor order

Sat, 27 Dec 2025 12:00:00 GMT

The following C++ code prints when run:

Base::Base
A::A
B::B
Derived::Derived
Derived::~
B::~
A::~
Base::~

The code:

struct Base
{
    Base() { std::cout << "Base::Base" << std::endl; }
    virtual ~Base() { std::cout << "Base::~" << std::endl; }
};

struct A
{
    A() { std::cout << "A::A" << std::endl; }
    ~A() { std::cout << "A::~" << std::endl; }
};

struct B
{
    B() { std::cout << "B::B" << std::endl; }
    ~B() { std::cout << "B::~" << std::endl; }
};

struct Derived final : public Base
{
    A a;
    B b;
    Derived() { std::cout << "Derived::Derived" << std::endl; }
    ~Derived() { std::cout << "Derived::~" << std::endl; }
};

int main()
{
    Derived derived;
    return 0;
}

From C++20 standard draft:

In a non-delegating constructor, initialization proceeds in the following order: (13.1) — First, and only for the constructor of the most derived class (6.7.2), virtual base classes are initialized in the order they appear on a depth-first left-to-right traversal of the directed acyclic graph of base classes, where “left-to-right” is the order of appearance of the base classes in the derived class base-specifier-list. (13.2) — Then, direct base classes are initialized in declaration order as they appear in the base-specifier-list (regardless of the order of the mem-initializers). (13.3) — Then, non-static data members are initialized in the order they were declared in the class definition (again regardless of the order of the mem-initializers). (13.4) — Finally, the compound-statement of the constructor body is executed. [Note: The declaration order is mandated to ensure that base and member subobjects are destroyed in the reverse order of initialization. —end note]

Invert any boolean condition

Sat, 27 Dec 2025 12:00:00 GMT

The negation of A || B is !A && !B
The negation of A && B is !A || !B

Links

See Boolean algebra, “De Morgan’s laws”: https://en.wikipedia.org/wiki/De_Morgan%27s_laws

An example

An example in C++ from the spdlog library, stdout_sinks-inl.h:

  // don't throw to support cases where no console is attached,
  // and let the log method to do nothing if (handle_ == INVALID_HANDLE_VALUE).
  // throw only if non stdout/stderr target is requested (probably regular file and not console).
  if (handle_ == INVALID_HANDLE_VALUE && file != stdout && file != stderr) {
      throw_spdlog_ex("spdlog::stdout_sink_base: _get_osfhandle() failed", errno);
  }

We group the 2nd and 3rd conditions into a new one:

if (handle_ == INVALID_HANDLE_VALUE && (file != stdout && file != stderr)) {

we apply the rule:

if (handle_ != INVALID_HANDLE_VALUE || !(file != stdout && file != stderr)) {

then apply the rule again on the 2nd part to get the inverted condition:

file == stdout || file == stderr

finally, we substitute it above:

if (handle_ != INVALID_HANDLE_VALUE || file == stdout || file == stderr) {

So the opposite of “if file is an invalid file and no console is attached” is “if file is a valid file or a console is attached”:

  # highlight-start
  // throw if handle_ is valid or a console is attached.
  if (handle_ != INVALID_HANDLE_VALUE || file == stdout || file == stderr) {
  # highlight-end
      throw_spdlog_ex("spdlog::stdout_sink_base: _get_osfhandle() failed", errno);
  }

Get windows error description from exception codes

Sat, 27 Dec 2025 12:00:00 GMT

$ certutil -error 0x5

0x5 (WIN32: 5 ERROR_ACCESS_DENIED) -- 5 (5)
Error message text: Access is denied.
CertUtil: -error command completed successfully.

You can also use decimal numbers, for example, the following are equivalent:

$ certutil -error 50
$ certutil -error 0x32

0x32 (WIN32: 50 ERROR_NOT_SUPPORTED) -- 50 (50)
Error message text: The request is not supported.

To view multiple possible error meanings from multiple header files, see the error lookup tool https://learn.microsoft.com/en-us/windows/win32/debug/system-error-code-lookup-tool

$ Err_6.4.5.exe c000021a
# for hex 0xc000021a / decimal -1073741286
STATUS_SYSTEM_PROCESS_TERMINATED                ntstatus.h
# {Fatal System Error}
# The %hs system process terminated unexpectedly with a
# status of 0x%08x (0x%08x 0x%08x).
# The system has been shut down.
# as an HRESULT: Severity: FAILURE (1), FACILITY_NULL (0x0), Code 0x21a
# for hex 0x21a / decimal 538
ERROR_ABIOS_ERROR                               winerror.h
# An error occurred in the ABIOS subsystem.
# 2 matches found for "c000021a"

Links

ThioJoe - “Never Wonder About Windows Errors Again”: https://www.youtube.com/watch?v=88a-cW9vNxY

Get a list of all shell built-ins

Wed, 05 Jul 2023 22:00:00 GMT

The help built-in command explains what a particular shell built-in does. The help menu of help says:

$ help --help
# or $ help help :)

help: help [-dms] [pattern ...]
    Display information about builtin commands.

    Displays brief summaries of builtin commands.  If PATTERN is
    specified, gives detailed help on all commands matching PATTERN,
    otherwise the list of help topics is printed.

    Options:
      -d        output short description for each topic
...

So, if you use the -d option to get a short description, and pass the '*' as a pattern to get information for all topics, you get a list with all the shell built-in commands and what they do:

$ help -d '*'

Shell commands matching keyword `*'

% - Resume job in foreground.
(( ... )) - Evaluate arithmetic expression.
. - Execute commands from a file in the current shell.
: - Null command.
[ - Evaluate conditional expression.
[[ ... ]] - Execute conditional command.
alias - Define or display aliases.
bg - Move jobs to the background.
bind - Set Readline key bindings and variables.
break - Exit for, while, or until loops.
builtin - Execute shell builtins.
caller - Return the context of the current subroutine call.
case - Execute commands based on pattern matching.
cd - Change the shell working directory.
...
until - Execute commands as long as a test does not succeed.
variables - Common shell variable names and usage.
wait - Wait for job completion and return exit status.
while - Execute commands as long as a test succeeds.
{ ... } - Group commands as a unit.

How do you know if a command is a shell built-in?

You can use the type shell built-in:

$ type cd

cd is a shell builtin

$ type cp

cp is /usr/bin/cp
# cp is not a shell built-in

What are shell built-in commands anyway?

TL;DR:

Builtin commands are contained within the shell itself.

Inspect text that contains control characters in bash

Wed, 05 Jul 2023 17:00:00 GMT

My use case: See what the IFS environment variable contains.

Some context: The IFS is the input field separator environment variable and it’s used by programs such as awk to separate the fields (columns) of a line.

For example, if a line contains the text 1 2 3 and ${IFS} is set to the space character, awk will give you 3 columns 1, 2, and 3. If IFS is set to tab, though, awk will give you only 1 column, a column that contains all the numbers 1 2 3.

It’s also used by the read shell built-in, a utility that’s used to read lines and fields from files and from the standard input (see help read).

It’s also worth noting that IFS can contain a list of characters, not only one character.

See Bourne shell reserved variables if want to know learn more environment variables.

So, let’s see what the IFS variable contains in my terminal:

$ echo "${IFS}"

# no visible out

I get no visible output, but if I pipe the result to the xxd command:

$ echo "${IFS}" | xxd

00000000: 2009 0a0a                                 ...
# xxd output explanation:
# line address: | hex bytes in pairs of 2 by default | textual representation

I get 4 characters. There’s an extra new line character there 0a because echo adds an extra new line at the end of the output.

We can use the -n option of echo to not append a newline (type help echo for more options).

$ echo -n "${IFS}" | xxd

00000000: 2009 0a                                   ..

It seems that my IFS environment variable consists of 3 characters:

0x20 the space
0x09 the tab (horizontal)
0x0a the line feed, aka new line, or \n, or LF.

The Wikipedia page List of Unicode characters will probably serve you well if you want to translate hex codes to Unicode characters.

Text to UTF-8 bytes

You can also use xxd to see the UTF-8 encoding (Unicode) of some text.

In the following example, I print my name (Mark) in English:

$ echo -n "Mark" | xxd

00000000: 4d61 726b                                Mark

Not too exciting, I gave it 4 characters, I got back 4 bytes.

But if I type my name in Greek:

$ echo -n "Μάρκος" | xxd

00000000: ce9c ceac cf81 ceba cebf cf82            ............

I get the UTF-8 encoding; how the text in stored in the file as bytes. I gave it “6” characters, I got back 12 bytes.

A “quick” way to verify this is with JavaScript’s encodeURIComponent method:

$ node
Welcome to Node.js vx.x.x.
Type ".help" for more information.
> encodeURIComponent("Μάρκος").split("%").join(" ").toLowerCase();

' ce 9c ce ac cf 81 ce ba ce bf cf 82'

Bash default encoding

If you want to see what’s the default encoding in your shell, search for environment variables that start with LC* (stands for locale) or LANG*:

$ printenv | grep -iE 'LC|LANG'

LANG=en_US.UTF-8

The printenv command above retrieves the values of all environment variables. The | (pipe) symbol redirects the printenv output to the grep command.

The grep command is used to search for patterns in text. The options used are:

-i: Performs a case-insensitive search.
-E: Enables extended regular expressions for pattern matching.

The pattern ‘LC|LANG’ specifies the search criteria. It looks for lines that contain either “LC” or “LANG”.

Join the lines of two files based on a common column

Tue, 04 Jul 2023 16:00:00 GMT

Consider the following files:

# file 1:
A    1
B    3
C    9
D    4

# file 2:
E    twelve
B    three
C    nine
A    one

Column 1 contains the key in both files, which is a capital letter.
Column 2 in file 1 contains a number
Column 2 in file 2 contains the same number in text format this time.

If we want a new file with:

the key in column 1,
the number in column 2 (from file 1),
and the number in text in column 3 (from file 2),

we can use the following join command:

$ join <(echo 'A    1
B    3
C    9
D    4'  | sort) <(echo 'E    twelve
B    three
C    nine
A    one'  | sort)

A 1 one
B 3 three
C 9 nine

Note that the inputs of join should be sorted, you’ll get an error by default. Also, the <( COMMAND ) syntax is called process substitution, and you can see it as a temporary file that’s not saved anywhere.

I would like to join on specific columns, though. When I typed join --help I noticed the following:

The default join field (i.e column) is the first, delimited by blanks.

You can specify on which column you want to join each of the two files with the -1 and -2 options. With -1 you specify the column number that contains the key from file 1, and, with -2, unsurprisingly, you specify the column number that contains the key from file 2.

For example, if the 2nd file had it’s columns in a different order, and the key was in the 2nd column:

# file 2:
twelve E
three B
nine C
one A

We could still join the lines of the 2 files based on the letters (key), but based on the second column for file 2 this time, with the following command:

$ join -2 2 <(echo 'A 1
B    3
C    9
D    4'  | sort) <(echo 'twelve E
three B
nine C
one A'  | sort -k 2)

A 1 one
B 3 three
C 9 nine

-2 2 means that the 2nd column from file 2 contains our key.

Note that sort -k 2 means “sort by the second column”.

SQL joins

You may wonder if you can join files with the join command, and get a similar behavior to SQL joins. In SQL, some common types of joins are the inner join, the left join, the right join, and the full join. It seems that you can perform all those joins with the join command!

For a TL;DR see the code blocks at the end of full join.

Also, if you need to refresh your SQL joins (I did), see the YouTube video SQL Joins Explained.

Inner join

By default, and in the previous examples, we got an inner join. Let’s review the 1st example again:

# file 1:
A    1
B    3
C    9
D    4

# file 2:
E    twelve
B    three
C    nine
A    one

# command
$ join <(echo 'A    1
B    3
C    9
D    4'  | sort) <(echo 'E    twelve
B    three
C    nine
A    one'  | sort)

# result
A 1 one
B 3 three
C 9 nine

The result is an inner join by default, because the row with key D from file 1 doesn’t have a matching row in file 2, and it’s not in the final result. Additionally, the row with key E from file 2 does not have a matching row in file 1, and it’s also omitted from the final result.

Left join

If we want to perform a left join, meaning that we want all the rows from file 1, and for those rows from file 1 that don’t have a matching row from file 2 (the one with key D), we will show null, we can use the -a FILE_NUMBER option.

The join --help says about the -a option:

  -a FILENUM
    also print unpairable lines from file FILENUM, where
    FILENUM is 1 or 2, corresponding to FILE1 or FILE2

So, we can use -a 1 to perform a left join (and include the unpairable lines from file 1 in the output):

$ join -a 1 <(echo 'A    1
B    3
C    9
D    4'  | sort) <(echo 'E    twelve
B    three
C    nine
A    one'  | sort)

# result
A 1 one
B 3 three
C 9 nine
D 4

The result can be improved though. I would like an output like the following:

A 1 one
B 3 three
C 9 nine
D 4 null

We can achieve the above with 2 extra join options (output from man join follows):

-e <EMPTY>, replace missing input fields with EMPTY.
-o <FORMAT>, obey FORMAT while constructing output line. FORMAT is one or more comma or blank separated specifications, each being FILENUM.FIELD

FILENUM can be 1 or 2, and FIELD is the column number (again 1 or 2, we have 2 columns in each file).

So, to improve the output of the left join, we can use the following command:

$ join -a 1 -e null -o '1.1,1.2,2.2' <(echo 'A    1
B    3
C    9
D    4'  | sort) <(echo 'E    twelve
B    three
C    nine
A    one'  | sort)

A 1 one
B 3 three
C 9 nine
D 4 null

That’s better!

Right join

And if we want to perform a right join, where we won’t have the row with key D, because it’s not present in file 2 at all, and the row with key E won’t have a number (from file 1) because it’s not present in file 1, we can use the following command:

$ join -a 2 -e null -o '2.1,1.2,2.2' <(echo 'A    1
B    3
C    9
D    4'  | sort) <(echo 'E    twelve
B    three
C    nine
A    one'  | sort)

A 1 one
B 3 three
C 9 nine
E null twelve

Note that the above is a right join this time (on file 2), so we want the 2.1 instead of the 1.1 as the key in the -o option.

Full join

In a full (outer) join, you want all the keys from both files, and wherever you don’t have a value, you print null.

This is the closest we can get with a custom output format:

$ join -a 1 -a 2 -e null -o '1.2,2.1,1.2,2.2' <(echo 'A    1
B    3
C    9
D    4'  | sort) <(echo 'E    twelve
B    three
C    nine
A    one'  | sort)

1 A 1 one
3 B 3 three
9 C 9 nine
4 null 4 null
null E null twelve

Not so pretty… But then I noticed the following:

If FORMAT is the keyword ‘auto’, then the first line of each file determines the number of fields output for each line.

I don’t fully get it right now, but if you use -o auto it does what we want (in this simple example…):

$ join -a 1 -a 2 -e null -o 'auto' <(echo 'A    1
B    3
C    9
D    4'  | sort) <(echo 'E    twelve
B    three
C    nine
A    one'  | sort)

A 1 one
B 3 three
C 9 nine
D 4 null
E null twelve

In fact, you can use -o auto for left join:

$ join -a 1 -e null -o auto <(echo 'A    1
B    3
C    9
D    4'  | sort) <(echo 'E    twelve
B    three
C    nine
A    one'  | sort)

A 1 one
B 3 three
C 9 nine
D 4 null

and for right join:

$ join -a 2 -e null -o auto <(echo 'A    1
B    3
C    9
D    4'  | sort) <(echo 'E    twelve
B    three
C    nine
A    one'  | sort)

A 1 one
B 3 three
C 9 nine
E null twelve

And, in order to have all the joins in one place, this is again the command for the inner join:

$ join <(echo 'A    1
B    3
C    9
D    4'  | sort) <(echo 'E    twelve
B    three
C    nine
A    one'  | sort)

A 1 one
B 3 three
C 9 nine

Get specific rows from a file

Thu, 29 Jun 2023 18:00:00 GMT

I wanted to update some python dependencies with pip, and I got this output when I run pip list --outdated:

$ pip list --outdated

Package    Version Latest Type
---------- ------- ------ -----
fonttools  4.33.3  4.40.0 wheel
pip        22.1    23.1.2 wheel
setuptools 58.1.0  68.0.0 wheel

And I thought: “Hm, it would be nice if I could print from the 3rd line and below, I don’t want the table header and separator in my output. This way, I can grab the package names with ease”.

You can print specific lines with awk and the NR built-in variable. NR contains the row number (aka line), which is an index that starts from 1:

$ pip list --outdated |
  awk '{ if (NR > 2) print $0 }'

fonttools  4.33.3  4.40.0 wheel
pip        22.1    23.1.2 wheel
setuptools 58.1.0  68.0.0 wheel

print $0 means print all the fields (columns) in the row. $1 refers to the first field, $2 to the second, and so on. awk is build for that kind of stuff.

And the complete command, where I upgrade the dependencies (use it at you own risk, check first which dependencies you really want to upgrade!):

$ pip list --outdated |
  awk '{ if (NR > 2) print $0 }' |
  grep -v pip |
  cut --delimiter ' '  --field 1 |
  xargs pip install --upgrade

grep -v pip removes pip from the results ( -v, --invert-match select non-matching lines) because I didn’t want to upgrade pip this way (I got an error when I did ERROR: To modify pip, please run the following command:..).
With cut --delimiter ' ' --field 1 I get the 1st column from the output which is the package name. The default delimiter for cut I believe is the tab character, that’s why I use a space as a delimiter.
With command | xargs other-command, if the command output is, for example:

# output of "command"
1
2
3

You transform it to:

$ other-command 1 2 3

In other words, you pass each line from the output of the previous command as a positional argument to the next command (the one that xargs calls), and execute the next command (other-command in this example).

And if you want only a specific row, for example the 2nd row:

$ echo -e '3\n6\n1' | awk '{ if (NR == 2) print $0 }'

6

echo -e means enable interpretation of backslash escapes, if you don’t use it, it will print:

$  echo '3\n6\n1'

3\n6\n1

Use the paste command if you’re feeling adventurous

A less robust way would be to use the paste command with the --serial option and then the cut command:

# doesn't work pip list -o, only with echo for some reason
# probably pip list -o has some funny characters?
$ echo 'Package    Version Latest Type
---------- ------- ------ -----
fonttools  4.33.3  4.40.0 wheel
pip        22.1    23.1.2 wheel
setuptools 58.1.0  68.0.0 wheel' |
  paste --serial --delimiters '*' |
  cut -d '*' -f 3-  |
  tr '*' '\n'

# result
fonttools  4.33.3  4.40.0 wheel
pip        22.1    23.1.2 wheel
setuptools 58.1.0  68.0.0 wheel

In the example above, I print the text directly into the console with echo because pip list --outdated doesn’t really work here with this method, for some reason I don’t care to investigate.

To understand how paste works, see a note of mine on how to split even and odd lines into two columns with paste.

cut -f 3- means get from the 3 column until the last column (see man cut for more stuff the -f option can do). And, if you want a specific row, for example the 3rd:

$ pip list -o |
  paste --serial --delimiters '*' |
  cut -d '*' -f 3

I’m using ’*’ as delimiter; find a char that is not part of your data/output/files.

Split the even and the odd lines of one column into two separate columns

Tue, 27 Jun 2023 16:00:00 GMT

Let’s start with the requirement in the title. You have this:

# file 1

1
A
2
B
3
C

And you want this:

1 A
2 B
3 C

In other words, you want odd rows (#1, #3, #5, etc) in column 1 and even rows (#2, #4, #6, etc) in column 2.

You can use the paste command like this:

$ paste --serial --delimiters '\t\n' <(echo '1
A
2
B
3
C')

1       A
2       B
3       C

That seemed like a hack to me. Let’s break down the command above step by step:

<( command_to_run ): This is called process substitution. It allows the output of a command to be treated as a file-like object. In this case, the output of the echo command will be used as input “file” for the paste command.
paste: This is a command that merges lines from multiple files. In this case, it will merge lines from the file-like object provided by the process substitution, we have only 1 file.
--delimiters '\t\n': This option specifies the delimiters to be used between merged lines. In this case, it alternates between a tab character (\t) and a newline character (\n) as delimiters.
--serial: I left this option last because it’s a bit tricky. It tells paste to paste one file at a time instead of in parallel. Lets see some examples first.

# file 1
1
2
3

# file 2
a
b
c

Default paste:

$ paste <( echo -e '1\n2\n3' ) <( echo -e 'a\nb\nc' )

1       a
2       b
3       c

With serial option:

$ paste --serial <( echo -e '1\n2\n3' ) <( echo -e 'a\nb\nc' )

1       2       3
a       b       c

# or with even more rows for file 1:
$ paste --serial <( echo -e '1\n2\n3\n4\n5' ) <( echo -e 'a\nb\nc' )

1       2       3       4       5
a       b       c

Also, an example if the files had 2 columns:

# file 1
1 Name1
2 Name2
3 Name3

# file 2
a 10
b 20
c 39

$ paste  <( echo -e '1 Name1\n2 Name2\n3 Name3')  <( echo 'a 10\nb 20\nc 39' )

1 Name1 a 10
2 Name2 b 20
3 Name3 c 39

And with 2 columns and --serial:

$ paste --serial <( echo -e '1 Name1\n2 Name2\n3 Name3' ) <( echo -e 'a\nb\nc' )

1 Name1 2 Name2 3 Name3
a       b       c

So, when you run paste, by default:

It will go to the 1st line of the resulting file, and it will print the whole 1st line from file 1.
In the same line, it will add a delimiter and it will print the whole 1st line from file 2.
In the same line, it will add a delimiter and it will print the whole 1st line from file 3, and so on, for each file.
It will go the 2nd line of the resulting file, and it will print the whole 2nd line from file 1.
In the same line, it will add a delimiter and it will print the whole 2nd line from file 2.
In the same line, it will add a delimiter and it will print the whole 2nd line from file 3, and so on, for each file.
And so on, for each line.

Hence the merges lines from multiple files you saw before.

If you run paste with the --serial option, it will do the following:

It will go to the 1st line of the resulting file, and it will print the whole 1st line from file 1.
In the same line, it will add a delimiter and it will print the whole 2nd line from file 1.
In the same line, it will add a delimiter and it will print the whole 3rd line from file 1, and so on, for each line of file 1.
It will go the 2nd line of the resulting file, and it will print the whole 1st line from file 2.
In the same line, it will add a delimiter and it will print the whole 2nd line from file 2.
In the same line, it will add a delimiter and it will print the whole 3rd line from file 2, and so on, for each line of file 2.
And so on, for each file.

In other words, with the --serial option, the paste command will print all the lines of each file, separated by the delimiter, in the same line. And each file will be in a different line.

Without the --serial option, the paste command will print each line of each file in order, separated by the delimiter, in the same line. And each line of the input files will be in a different line.

Let’s see some examples with the --delimiters option to understand what’s going on:

$ paste --delimiters '^' <( echo -e '1\n2\n3' ) <( echo -e 'a\nb\nc' )

1^a
2^b
3^c

Ok, it replaces the default tab delimiter with the ^ character, this makes sense.

What if we add another character in the --delimiters because man curl says that: -d, --delimiters=LIST, reuse characters from LIST instead of TABs

$ paste --delimiters '^#' <( echo -e '1\n2\n3' ) <( echo -e 'a\nb\nc' )

1^a
2^b
3^c

So, nothing changed here. That is because it did not have the need for an additional delimiter, the files were only two.

But if you use the --serial option, there is room for 2 delimiters because each file has 3 lines:

$ paste --delimiters '^#' --serial <( echo -e '1\n2\n3' ) <( echo -e 'a\nb\nc' )

1^2#3
a^b#c

And now let’s add again the tab and the new line as delimiters, the same as the original solution:

$ paste --delimiters '\t\n' --serial <( echo -e '1\n2\n3' ) <( echo -e 'a\nb\nc' )

1       2
3
a       b
c

Not the result we wanted to achieve in the title, but, in our defense, the input is different! And I also hope that the original solution makes more sense now.

If you use serial, and you limit the input to 1 file, you’ll get the same result:

paste --delimiters '\t\n' --serial <( echo -e '1\na\n2\nb\n3\nc' )

1       a
2       b
3       c

Quiz

You have the following file:

1
a
A
2
b
B
3
c
C

How do you transform it to:

1 a A
2 b B
3 c C

Solution:

# notice the x2 spaces in the delimiters
paste --delimiters '  \n' --serial \
  <( echo -e '1\na\nA\n2\nb\nB\n3\nc\nC' )

1 a A
2 b B
3 c C

Use awk if you want more flexibility

You can achieve the same result with awk and the NR built-in variable (it stores the row number):

$ awk '{ if (NR % 2) { printf "%s", $1 } else { printf "\t%s\n", $1}}' \
  <(echo "1
A
2
B
3
C")

1       A
2       B
3       C

The above says, if the row number is odd, print the first column, else (the row number is even), print a tab character, the first column, and a newline character (change the line). Repeat this for every line in the file.

Get specific columns from multiple files and paste (combine) them into a new file

Sun, 25 Jun 2023 16:00:00 GMT

Assuming that the first file shows the block size of a file in column 1 and the file path in column 2:

# file 1
$ du -sh ~/bin/*

0       /c/Users/asdf/bin/abswin
0       /c/Users/asdf/bin/abwin
37M     /c/Users/asdf/bin/BrowserStackLocal.exe
0       /c/Users/asdf/bin/cloc
0       /c/Users/asdf/bin/curltime
960K    /c/Users/asdf/bin/cvdump.exe

And file 2 has the file size in column 5:

# "file" 2
$ ls -lha ~/bin/* | awk '{print $5}'

lrwxrwxrwx 1 asdf 197121   27 Nov 21  2022 /c/Users/asdf/bin/abswin -> /c/abs-from-laragon/abs.exe
lrwxrwxrwx 1 asdf 197121   56 Nov 20  2022 /c/Users/asdf/bin/abwin -> /c/laragon/bin/apache/httpd-2.4.47-win64-VS16/bin/ab.exe*
-rwxr-xr-x 1 asdf 197121  37M Dec 16  2022 /c/Users/asdf/bin/BrowserStackLocal.exe*
lrwxrwxrwx 1 asdf 197121   66 Oct 28  2022 /c/Users/asdf/bin/cloc -> /c/Users/asdf/Desktop/dev/personal-projects/shell-scripts/clock.sh*
lrwxrwxrwx 1 asdf 197121   69 Nov  2  2022 /c/Users/asdf/bin/curltime -> /c/Users/asdf/Desktop/dev/personal-projects/shell-scripts/curltime.sh*
-rwxr-xr-x 1 asdf 197121 957K May  3 14:40 /c/Users/asdf/bin/cvdump.exe*

If you don't have files in your ~/bin directory, you can grab some files from the /bin directory. For example, replace du -sh ~/bin/* with du -sh /bin/* | head -n 5. The bin folder of your home directory (the ~ character means home directory), is in the PATH variable, so you can refer and execute them from any folder.

Let’s assume that I want the following output. First, the column 2 of file 1 (name), second, the column 1 of file 1 (size), and finally, column 5 of file 2 (size):

/c/Users/asdf/bin/abswin        27      0
/c/Users/asdf/bin/abwin 56      0
/c/Users/asdf/bin/BrowserStackLocal.exe 37M     37M
/c/Users/asdf/bin/cloc  66      0
/c/Users/asdf/bin/curltime      69      0
/c/Users/asdf/bin/cvdump.exe    957K    960K

Use the cut --fields LIST or awk '{print $5}' command to print specific columns from a file, and use the paste utility to combine them into a new file. See the following command for example:

$ paste \
  <(du -sh ~/bin/* | cut -f 2) \
  <(ls -lha ~/bin/* | awk '{print $5}') \
  <(du -sh ~/bin/* | cut -f 1)

/c/Users/asdf/bin/abswin        27      0
/c/Users/asdf/bin/abwin 56      0
/c/Users/asdf/bin/BrowserStackLocal.exe 37M     37M
/c/Users/asdf/bin/cloc  66      0
/c/Users/asdf/bin/curltime      69      0
/c/Users/asdf/bin/cvdump.exe    957K    960K

In the command above, I use process substitution <(...) to generate the output of the du, cut, and awk commands as input to the paste utility. See the process substitution as something that creates a temporary file that doesn’t get saved anywhere. The du command retrieves the disk usage of files, the cut command extracts specific fields, and the awk command prints the fifth field.

I also use linux pipes, we combine single letter options, e.g. -sh = -s + -h, the -h option in all the previous commands means human readable, the -f option means field (or column). If you don’t understand a command, copy and paste it in the Explain shell web application.

Web scraping in the console

Mon, 02 Nov 2020 15:00:00 GMT

This post contains a collection of code snippets you can paste into the browser’s console and get useful data back. It’s like scraping a web page, but instead of doing it inside a Node script, you do it in the browser’s console. Because you shouldn’t run code that you don’t understand in your console, I explain how most of the snippets work. As a result, you may learn some JavaScript tricks you haven’t seen before, learn how to accomplish basic tasks without using a framework/library (not optimized for performance though), and extract some useful data from the pages you visit. Before we see the code, I want to explain some syntax you’ll encounter throughout the snippets.

In the browser, the $ function is an alias for document.querySelector. For example, if you run $('body') in the console, you’ll get back the body element which is the same as running document.querySelector('body'). The $$ is the equivalent for document.querySelectorAll. I use both of those aliases to save some space in the code snippets.

The $$ returns a NodeList that lacks many of the useful array methods such as map and filter. For that reason, I use the Array.from(notAnArray) method to transform it into an array.

I also use the Set constructor new Set(array) that returns a Set object that contains all the unique values inside an array. Here, we face the same issue with the $$ alias, so I transform the Set to an array with the Array.from method.

Show outlines to find overflows

Find which elements cause overflows by adding a red outline to all the body elements. This can also help you understand how inline elements work.

Array.from($$("body *")).forEach((el) => {
  el.style.outline = "1px solid red";
});

Get all the different tags used in a page

It’s not the most exciting one, but you may discover new elements to use or at least see what elements other sites use.

Array.from(
  new Set(Array.from($$("body *")).map((el) => el.tagName))
).sort();

You get all the elements inside the body element with $$("body *") call that returns a NodeList, you make it an array, you transform that element array to an array with the tag names (strings), you keep only the unique tags with the Set constructor, and finally, you transform the Set to an array. The sort() call at the end sorts the array in alphabetical order.

Typography

Print a table with all the characters

See what characters a website uses. Use this if you want to adjust the font files by creating subsets to make sure they cover those characters. The final result is the end of the section.

You start by getting all the elements inside the body and filter out those elements that don’t contain human-readable text, for example, scripts, styles, iframes, etc.

Array.from($$("body *")).filter(
  (tagName) =>
    ![
      "defs",
      "style",
      "STYLE",
      "IFRAME",
      "path",
      "g",
      "svg",
      "SCRIPT",
      "NOSCRIPT",
      "",
    ].includes(tagName)
);

You transform the elements (with map) to an array that contains their inner texts, you keep only the values that are truthy with filter(Boolean)—this will remove empty strings (""), undefined, and more—and you transform it into an array of characters with the join("") array method. The final stage is to pass the output to Array.from(new set(output)).sort() that will return a sorted array with unique characters.

Array.from(
  new Set(
    Array.from($$("body *"))
      .filter(
        (tagName) =>
          ![
            "defs",
            "style",
            "STYLE",
            "IFRAME",
            "path",
            "g",
            "svg",
            "SCRIPT",
            "NOSCRIPT",
            "",
          ].includes(tagName)
      )
      .map((el) => el.innerText)
      .filter(Boolean)
      .join("")
  )
).sort();

Why do this instead of splitting the joined string with split("")? Because, this way, you preserve characters like emojis (and many more). split, on the other hand, breaks those characters apart and returns jibberish.

Transform the characters to code points and sort them, and finally, print the result. The result is an array with the characters, along with their Unicode hex numbers.

console.table(
  Array.from(
    new Set(
      Array.from($$("body *"))
        .filter(
          (tagName) =>
            ![
              "defs",
              "style",
              "STYLE",
              "IFRAME",
              "path",
              "g",
              "svg",
              "SCRIPT",
              "NOSCRIPT",
              "",
            ].includes(tagName)
        )
        .map((el) => el.innerText)
        .filter(Boolean)
        .join("")
    )
  )
    .sort()
    .map((char) => char.codePointAt())
    .sort((a, b) => a - b)
    .map((codePoint) => ({
      unicode: codePoint.toString(16).toUpperCase(),
      character: String.fromCodePoint(codePoint),
    }))
);

See what fonts a website uses

Or, more specifically, get the different values of the font-family CSS attributes of all body elements. You accomplish that with the help of the getComputedStyle method:

new Set(
  Array.from($$("body *")).map((el) => getComputedStyle(el).fontFamily)
);

In case you’re wondering, you cannot do the same with el.style because the CSS properties of that object are not populated from the stylesheets. el.style is used for setting properties with JavaScript. See all the differences between getComputedStyle and el.style.

Firefox developer tools do a much better job at this task with the Fonts tab, that’s inside the Inspector tab.

More typographic data

Get the font families and the different font sizes that are used in:

console.table(
  Array.from(
    new Set(
      Array.from($$("body *")).map((el) => {
        var { fontFamily, fontSize } = getComputedStyle(el);
        return JSON.stringify({ fontFamily, fontSize });
      })
    )
  )
    .map((uniqueKey) => JSON.parse(uniqueKey))
    .sort(
      (a, b) =>
        a.fontFamily.replace('"', "").toUpperCase().codePointAt() -
        b.fontFamily.replace('"', "").toUpperCase().codePointAt()
    )
);

The Set constructor finds the unique values from arrays that contain primitive values. In this case, we want both the family and the size of an element, so the first thought might be to create an object for each element and store that information there. If you do that, Set will not work because it will compare the objects by reference, not by the inner value to find if they are unique. For this reason, you serialize the object to a string with JSON.stringify and later transform it back to an object with JSON.parse (see the highlighted lines).

I have a post that tackles a similar issue which is the difference between deep/shallow copy and the assignment operator. It contains useful references to other resources to learn more about the subject.

Highlight characters from a specific script

A use case for me is that some Greek characters are identical to Latin, for example, Τ/T or O/Ο. This code helps me find these small mistakes I make while writing text.

var regEx = /\p{Script_Extensions=Latin}/gu;
Array.from($$("h1, h2, h3, h4, p"))
  .filter((el) => regEx.test(el.innerHTML))
  .forEach((el) => {
    el.innerText = el.innerText.replace(regEx, "$&\u0332");
  });

Regular expressions are not the most readable code in the world, but they have some cool features. One of them is the Unicode property escapes (e.g: /\p{property}/u). You can use them in regular expressions to find characters from a specific script, emojis, punctuation marks, and more—see the link for more properties. Don’t forget to add the Unicode flag (u) when you use Unicode property escapes. I’m also using $& in the string replace method to refer to the matched string.

I mark the characters with the mark Unicode character (combining low line u+0332). I initially thought to parse the HTML of the elements (not the innerText) with regular expressions and wrap the characters with <mark> elements, but, as it turns out, parsing HTML with regular expressions is probably a bad idea.

Test a font

See how a Google Font looks on a page. To do that, you create a style element, you add it inside the head element, and you use it.

var ibmSans = Object.assign(document.createElement("link"), {
  href: "https://fonts.googleapis.com/css2?family=IBM+Plex+Sans:ital,wght@0,400;0,700;1,400;1,700&display=swap",
  rel: "stylesheet",
});
document.head.appendChild(ibmSans);
// Use the font
Array.from($$("body, h1, h2, h3")).forEach((el) => {
  el.style.fontFamily = "'IBM Plex Sans', sans-serif";
});

In the previous example, I use a “trick” with Object.assign to create an element that looks like the React API for creating elements, for example:

// This
var redDiv = Object.assign(document.createElement("div"), {
  style: "background-color: red; width: 100px; height: 100px;",
});

// Looks similar to this React.createElement(tag, props, children)
var redDiv = React.createElement(
  "div",
  {
    style: {
      backgroundColor: "red",
      width: "100px",
      height: "100px",
    },
  },
  null
);

Add an external library

Vanilla JavaScript is cool, but sometimes you wish you had access to an external library to help you do the job. In the following example, you can add lodash with an external script from unpkg:

var script = document.createElement("script");
script.src = "https://unpkg.com/lodash@4.17.20/lodash.js";
script.onload = () => {
  console.log(_.map([1, 2, 3], (n) => n * 2));
};
document.head.appendChild(script);
// prints [2, 4, 6] when the script loads

The code above shows how to add an external script to a page with JavaScript. To add a different library from NPM, replace the :package from the following snippet with the package name, enter the URL in the browser, and unpkg will redirect you to the correct file. If not, you’ll have to browse the directory to find the file yourself, and in this case, don’t forget to remove the /browse/ from the URL:

unpkg.com/:package@latest
# And the original sample from unpkg:
unpkg.com/:package@:version/:file

Extract more CSS properties from a page

With the following snippet, you get all the different box shadows, but you can use it for any other CSS property you’re interested in.

Array.from(
  new Set(
    Array.from($$("body *")).map((el) => getComputedStyle(el).boxShadow)
  )
).sort();

Or create an object with the box shadows, colors, borders, and background images.

Object.entries(
  Array.from($$("body *")).reduce(
    (data, el) => {
      const style = getComputedStyle(el);
      data.boxShadows.add(style.boxShadow);
      data.colors.add(style.color);
      data.colors.add(style.backgroundColor);
      data.borders.add(style.border);
      data.borders.add(style.borderTop);
      data.borders.add(style.borderRight);
      data.borders.add(style.borderBottom);
      data.borders.add(style.borderLeft);
      data.backgroundImages.add(style.backgroundImage);
      return data;
    },
    {
      boxShadows: new Set(),
      colors: new Set(),
      borders: new Set(),
      backgroundImages: new Set(),
    }
  )
).map(([key, values]) => ({
  [key]: Array.from(values)
    .sort()
    .filter((cssVal) => cssVal && cssVal !== "none"),
}));

Show me the code

Show all the elements that are usually hidden. More specifically, this snippet shows all the head elements, and from the body, it shows the scripts, styles, and noscript elements.

document.querySelector("head").style.display = "block";
Array.from(
  $$("head > *, body script, body style, body noscript")
).forEach((el) => {
  var pre = document.createElement("pre");
  var code = document.createElement("code");

  pre.style.backgroundColor = "black";
  pre.style.color = "white";
  pre.style.padding = "1em";
  pre.style.marginBottom = "1.5em";
  pre.style.overflowX = "auto";
  pre.style.zIndex = 9999;
  code.style.backgroundColor = "inherit";
  code.style.color = "inherit";

  pre.appendChild(code);
  code.appendChild(el.cloneNode(true));
  el.insertAdjacentElement("afterend", pre);
  code.innerText = code.innerHTML;
});

In the previous snippet, you create a pre and a nested code element, and you style them. You also add the code in plain text inside the code element (see below how). The plan is to use them like this:

<pre>
  <code>
  // How to greet from the console.
  console.log("Hello world");
  </code>
</pre>

You use the insertAdjacentElement method to insert the <pre> right after the original element. The alternative is to get the parent node of the element with el.parentNode and append a child with the appendChild method. You set the code element’s inner text to its inner HTML which is the original (cloned) element’s HTML. If you don’t use the cloneNode() method to create copies of the original elements, the scripts and the styles will be rendered useless, and the page will not work as before.

Infinite scrollers, default styles on <pre> elements, and fixed elements can mess-up the result.

Get comments from inline scripts and styles

I’m not sure why you’d want to use this; maybe to read the licenses and the inner thoughts of your fellow developers? My favorite comments are the DO NOT CHANGE, all in caps of course.

Array.from(document.querySelectorAll("script, style"))
  .map((el) => el.textContent)
  .filter(Boolean)
  .reduce((result, code) => {
    var weirdRegEx =
      /(?<!:)(?<comment>\/{2}.*?)\n|(?<commentMulti>\/[*\s]*\*[\s\S]+?\*[*\s]*\/)/g;
    var comments = [];
    for (let match of code.matchAll(weirdRegEx)) {
      var comment = match.groups.comment || match.groups.commentMulti;
      comments.push(comment);
    }
    return result.concat(comments);
  }, []);

The regular expression for single-line comments gives many false positives though. For example, it may return base64 encoded data that match.

You could use either of textContent and innerText to get the text of styles and scripts, so it doesn’t matter which one you choose in this case. See all the differences between textContent and innerText.

This is a visualization of the regular expression for single-line comments created by the Regulex app. The (?<name>thing to name) creates a named capturing group that is easier to access via match.groups.name instead of match[1].

Regular expression visualization for single-line comments

// Single-line comment

And this is a visualization of the regular expression for multiline comments:

Regular expression visualization for multi-line comments

/*
  Multiline
  comment
*/

The dot special character . in regular expressions matches all characters except newlines. To match all characters including newlines, you can use [\s\S].

Get all the links from a page

Print the URLs and the text of the links in a table. Screen readers offer something similar with the rotor function:

console.table(
  Array.from(
    new Set(
      Array.from($$("a"))
        .map((link) =>
          link.href
            ? JSON.stringify({
                url: link.href,
                text: link.innerText,
              })
            : null
        )
        .filter(Boolean)
    )
  )
    .sort()
    .map((serialized) => JSON.parse(serialized))
);

If you don’t like the table from console.table, you can use a regular console.log. In Chrome’s console, you can resize the columns of the table and sort the data by column.

Images

Display only the images

Display only the image elements inside the page’s body—it removes the body content.

var images = document.querySelectorAll("img");
var body = document.querySelector("body");
body.innerHTML = "";
body.style.overflow = "auto";

images.forEach((img) => {
  var wrapper = document.createElement("div");
  wrapper.appendChild(img);
  body.appendChild(wrapper);
});

Show background images too and change styling

I have a more elaborate solution because the images many times have some default styling, such as absolute positions or weird widths. If you want a more consistent result, it’s better to create new image elements. This also creates image elements for the background images:

var images = document.querySelectorAll("img");
var backgroundImages = Array.from(document.querySelectorAll("body *"))
  .map((el) => getComputedStyle(el).backgroundImage)
  .filter((css) => css !== "none")
  .map((css) => ({
    // The .*? in the middle will match zero or more characters,
    // but as few as possible (non-greedy, greedy is the default).
    // If you use .* it will consume the ending quote and the URL
    //  will be invalid.
    src: css.match(/url\(["']?(.*?)["']?\)/)?.[1],
    alt: null,
  }));

var body = document.querySelector("body");
body.innerHTML = "";
body.style.overflow = "auto";

var elements = Array.from(images)
  .concat(backgroundImages)
  .filter(({ src }) => src)
  .map((img) => {
    var newImage = document.createElement("img");
    newImage.src = img.src;
    newImage.alt = img.alt || "";
    newImage.title = img.alt || "";
    newImage.style.display = "block";
    newImage.style.width = "auto";
    newImage.style.outline = "1px dashed red";
    return newImage;
  });

body.append(...elements);

I use append because I want to add multiple elements at once—see all the differences between append and appendChild. You can render the images in a mosaic layout (kind of) if you add some flex styles to the body element:

body.style.display = "flex";
body.style.flexWrap = "wrap";
body.style.alignItems = "flex-start";

Regular expression visualization for background images URLs.

Dark mode DIY

This trick uses the invert CSS filter to create a dark mode if the site doesn’t offer the option. I first saw it in a post on how to create a theme switcher by Heydon Pickering.

var filter = "invert(1)";
var root = document.querySelector("html");
root.style.backgroundColor = "white";
root.style.filter = filter;

var allElements = Array.from(document.querySelectorAll("body *"));
allElements.forEach((el) => (el.style.backgroundColor = "inherit"));

var media = Array.from(
  document.querySelectorAll('img:not([src*=".svg"]), video')
);
media.forEach((el) => (el.style.filter = filter));

Show duplicated ids

Ids on a page should be unique and it may be hard to notice if you don’t test with a tool like axe or Lighthouse. Duplicate ids are not always the result of your code; external libraries can cause them too. This snippet will help you identify this issue.

var elementsWithIds = Array.from($$("[id]"));
var ids = elementsWithIds
  .map((el) => el.getAttribute("id"))
  .filter(Boolean);
console.log({
  elementsWithIds,
  uniqueIds: new Set(ids),
  duplicateIds: ids.filter((el, index) => ids.indexOf(el) != index),
});

React’s useState hook details

Sun, 02 Aug 2020 00:00:00 GMT

In this post, you’ll see some details with the useState hook. More specifically, you’ll learn about:

The difference between lazy initialization and passing an initial value.
The difference between the functional and the regular setState.
The fact that the setState function remains the same during renders.

What’s the difference between lazy initialization and passing an initial value to `useState` hook?

Consider a typical useState hook (the initialValue represents a number or a string value):

const [value, setValue] = useState(initialValue);

If you declare a useState hook this way, React will use the initialValue for the first render and ignore it for subsequent renders. Ignoring the initial value on subsequent renders makes sense because you want to be able to change the state in your code.

If you calculate the initial value with an expensive function, though, React will still run the expensive function on every render. It will just discard the result. An example of what I’m talking about:

const [value, setValue] = useState(someExpensiveComputation(input));

This behavior isn’t specific to React. In the example above, you invoke a function that returns your initial state, so the JavaScript engine itself has to evaluate that function call to get the resulting value. If you don’t want to evaluate the expensive function on every render, you can use lazy initialization and pass a function as the first argument to the useState, as shown below:

const [value, setValue] = useState(() =>
  someExpensiveComputation(input)
);

This is different because you pass a function, not the result of a function call. You can also pass a function reference if your expensive function doesn’t need a parameter to make the code less verbose:

const [value, setValue] = useState(someExpensiveComputation);

You can observe this useState behavior in this CodeSanbox (comment / uncomment the lines).

What’s the difference between the functional and the regular `setState`?

In the following snippet, I use the useState hook to store and update a counter:

const [counter, setCounter] = useState(0);

The most straightforward way to update your counter state is to pass a value or an expression to the setState function (I call this regular form):

const [counter, setCounter] = useState(0);
// somewhere inside an event listener
setCounter(counter + 1);

The alternative is to pass a function (aka updater function) that has the previous state as a parameter and returns the new state (I call this functional form):

const [counter, setCounter] = useState(0);
// somewhere inside an event listener
setCounter((prev) => prev + 1);

Most of the times, these two forms of setState do the same thing. They start to differ if you call them inside an asynchronous function or multiple times inside the same function.

Let’s explain what happens when you call them inside an asynchronous function. That function can be a setTimeout, a promise.then block, or a requestAnimationFrame. For example:

const [counter, setCounter] = useState(0);
// somewhere inside an event listener
setTimeout(() => {
  setCounter(counter + 1);
}, 1000);

If you use the regular form to increase the counter inside the setTimeout, and then you increase the counter again by 1 in a synchronous function (or if you update 2 times quickly with the asynchronous function), the counter will remain at 1. This happens because the setTimeout callback captures the value of the counter at the time you create it. So it sees setCounter(0 + 1) and not setCounter(currentValue + 1). An important detail to remember here is that we get a new counter and function on every render. The asynchronous function closes over the counter value the time you create it. If you update the state with the asynchronous function, and then you update it again with a synchronous function, the asynchronous function won’t see the update. This is because the counter variable that was updated is not the same variable the asynchronous function sees.

On the other hand, if you use the functional form, the counter will increase 2 times and you’ll get a 2 as a result. The functional form doesn’t capture a specific value for the state but gets the previous value when it’s time to update the state.

The same thing happens if you call the setState multiple times inside the same method (see at the end of the chapter in the previous link). The functional form will update the state multiple times but the regular form only once. See also a relevant StackOverflow question.

Check a CodeSanbox that shows the difference between the functional and the regular setState.

The `setState` function from `useState` doesn’t change during renders

You saw in the previous section that the first variable useState returns (the counter) is different on every render. This is not true for the updater function (e.g. setCounter) which remains the same between renders. We can validate this with the following quote that comes from the Hooks reference in the React docs:

React guarantees that setState function identity is stable and won’t change on re-renders. This is why it’s safe to omit from the useEffect or useCallback dependency list.

The same is true for the dispatch function from the useReducer hook.

In addition to the dependency list part, this also means that you don’t have to wrap this function in a useCallback hook when you pass it down from a parent to a child component (to prevent the child from re-rendering if the parent component renders).

In other words, you don’t have to worry that the identity of the function will change between renders, as it changes when you pass to a child component a function you create inside the render method of the parent component.

What’s the difference between using a ref and a variable to keep track of a value in React?

Wed, 10 Jun 2020 20:00:00 GMT

Sometimes, you want to keep track of a value in React, but you don’t want to store it in the state because it will cause extra renders when it changes. The recommended method (for a functional component) is to use the useRef hook to store the value there.

In the following example, I want to run the effect when the state changes, but not the first time the component mounts. For this reason, I use a firstTime ref that initially is set to true and run the effect only when the firstTime.current is false:

import React, { useState, useEffect, useRef } from "react";

const Component = () => {
  const [state, setState] = useState(0);
  const firstTime = useRef(true);

  useEffect(() => {
    if (!firstTime.current) {
      // Run the effect.
    } else {
      firstTime.current = false;
    }
  }, [state]);

  return <div></div>;
};

export default Component;

The syntax is a bit ugly because you have to use the .current property, and you might be tempted to use a regular variable instead of the useRef hook, as shown in the next example.

import React, { useState, useEffect } from "react";

let firstTime = true;

const Component = () => {
  const [state, setState] = useState(0);

  useEffect(() => {
    if (!firstTime) {
      // Run the effect.
    } else {
      firstTime = false;
    }
  }, [state]);
  return <div></div>;
};

export default Component;

Differences

As long as the component is a singleton—meaning that you use only one instance of the component in your application—both methods do the same thing. Multiple instances, though, which is really common, share the same variable! As a result, you should avoid using a regular variable because it can get pretty bad. The following example illustrates the problem:

src/components/Counter.jsx

import React, { useState } from "react";

let counterOutside = 0;
const Counter = () => {
  const [counter, setCounter] = useState(0);
  console.log(counterOutside);
  return (
    <p>
      The counter is {counter}{" "}
      <button
        onClick={() => {
          setCounter(counter + 1);
          counterOutside = counterOutside + 1;
        }}
      >
        +
      </button>
    </p>
  );
};

export default Counter;

src/App.jsx

// ....
<div>
  <Counter />
  <Counter />
</div>

If you press the button inside one counter, it prints “1 1” where it should print “1 0”, “2, 2” where it should print “2 0”, and so on.

Creating font subsets

Wed, 06 May 2020 23:00:00 GMT

Two of the most important parts of a web font are the characters and the OpenType features. Characters are the letters of the English alphabet (A-Z, a-z), Arabic numbers (0-9), punctuation (comma, spaces, etc). The Unicode standard organizes the characters in blocks based on their functionality and the language they support. Each character in Unicode is mapped to a hex number.

The OpenType features, on the other hand, may include variations of those characters (aka glyphs) that are useful in different contexts. If we take the numbers, for example, they can be superscripts (x²), fractions (1/2), slashed zeros to separate them from the capital O, or old-style numbers (123456789) that are used to blend numbers inside text.

Why bother with font subsets.

Subsetting fonts is the process of taking a large font file as input and creating other smaller files, with fewer characters or OpenType features. There are several reasons you may want to subset a font file:

You want to use advanced typographic features or characters that are not included in the Google fonts subsets—see what the Google fonts subsets include in a later section. On the other hand, you may want to use fewer features or characters from the original font file.
You want to create small subsets for two-stage font loading to improve the performance of your application. The goal of this method is to reduce or eliminate the time the browser shows the fallback fonts or the invisible fonts. You can achieve that by loading a small file first, and, at some later point, by swapping it with a richer font file.
You want to split a large font file into script (or language) specific subsets to improve performance without getting overly technical with two-stage font loading. Google already does this, but you may want to create your own, custom subsets, or create subsets for a font that’s not on Google Fonts.
You want to use the latest version of a typeface, and you don’t want to wait until Google updates it.

Subsetting is probably more useful for free fonts because paid fonts may require to embed them in a specific way in your site (through a CDN), or the license you purchase may not permit subsetting.

In the next section, you’ll see what tools you can use to subset your fonts.

Available Tools

To create subsets, you can use online tools like the web font generator from Font Squirrel, but here I’ll show you how to use pyftsubset. pyftsubset is a popular command-line tool written in Python that is part of the fonttools. Apparently, Google Fonts use this tool to create their subsets, but, because I’m not familiar with Python, I don’t know if this is still true. Another useful tool is glyphhanger (it uses pyftsubset under the hood) that has a really cool feature that can crawl your site and show you what characters you use. You can use glyphhanger to subset your fonts, but I prefer pyftsubset because you can select what OpenType features you want.

To use glyphhanger —which is an NPM package by the way— check the project’s README on GitHub or a post by Zach Leatherman. I will use glyphhanger in one of the following examples.

Let’s see now how you can install the fontools and pyftsubset:

Install fonttools

To use pyftsubset, you need Python and the Python package manager (pip). Later versions of Python include pip, so you don’t have to install them separately. When you install Python, don’t forget to add it to the path (Windows installer gives you this option) because you want to use pip from the command line to install the required packages. To check if you installed them, run python --version and pip --version in your command line; both tools should print their versions.

Having installed Python and pip, you’ll use pip to install the following Python packages:

pip install fonttools brotli zopfli

Let’s see what those packages do:

You install the fonttools because you want to use the pyftsubset tool to subset your fonts.
You want Brotli to compress .woff2 files, which is the format used by modern browsers. You don’t have to run Brotli separately; it’s used by pyftsubset.
zopfli is used to compress .woff files, which is a format used by older browsers. zopfli is not the default option for compressing woff files, and, as a result, it’s optional. If you want to use it, you add the --with-zopfli option in your pyftsubset commands.

To subsets your fonts, you’ll want to know what characters and OpenType features they contain. Luckily, there’s an excellent online tool that can help you with that.

What the Google font subsets include

Let’s now see what the subsets from Google fonts include. You’ll use an online tool by Roel Nieskens called wakamaifondue (What can my font do). Go to the Google fonts interface and select the regular version of Work Sans. To download the font file, go to the embed code that Google gives you, copy the URL from the href attribute of the <link>, and open it in your browser. To save you some time, this is the URL I’m talking about:

https://fonts.googleapis.com/css2?family=Work+Sans&display=swap

This URL contains a CSS file that has the font faces for the subsets. If you open the Latin subset (the URL from the src descriptor) in your browser, you’ll start downloading that file. This is the URL of the font file at the time of writing:

https://fonts.gstatic.com/s/worksans/v7/QGY_z_wNahGAdqQ43RhVcIgYT2Xz5u32K0nXBi8Jpg.woff2

Download the file above and drop it in wakamaifondue. The app tells me that the Latin subset has 225 characters and 9 layout features.

The Work Sans in the wakamaifondue interface

OpenType features

kern, liga, calt, rvrn, locl: Browsers enable those features by default. kern adjusts the distance between glyphs, liga and calt transform some glyph combinations to single glyphs, rvrn is specific to variable fonts, and locl is script related.
cswh: Contextual swashes (Reality → Reality) are used for decoration.
frac (fractions: 5/8 → 5/8), numr (numerators: 123 → 123), and dnom (denominators: 123 → 123) are all different ways to display numbers.

A Google font subset with 9 OpenType features is not that usual; most times you’ll see subsets with 4 features.

Now do the same thing for the complete font file that you can find in the GitHub page of Work Sans. The app tells me that this file has 750 characters and 38 layout features. Although 750 compared to 225 characters seems a big difference, most of the characters the Latin subset is missing will be inside the other subsets (see the CSS file with the font faces). So let’s focus first on the OpenType features, and name some important that are not in the subsets:

onum, pnum: Old-style and proportional numbers are used to blend numbers inside text: 365 days → 365 days.
smcp, c2sc: Small caps are used for abbreviations and acronyms, and have the same purpose as old-style numbers which is to blend capital letters inside text: HTML document → HTML document.
subs, sups: Superscripts and subscripts for numbers.
dlig: Discretionary ligatures are used for decoration. You can use them together with contextual swashes to create an interesting logo for example: station → station.
salt or ss01 to ss05: Stylistic alternates that are variations of existing characters: R → R.

For more information about the OpenType features and how to use them in your code, check an interactive post by Tim Brown called Caring about OpenType features. I use a CSS package I made for that reason.

Characters

You saw what OpenType features the Google fonts subsets include and what features they’re missing; let’s now focus on the characters. Open again the CSS file that contains the font faces, and, this time, pay attention to the unicode-range descriptor:

/* latin */
@font-face {
  unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6,
    U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2191, U+2193,
    U+2212, U+2215, U+FEFF, U+FFFD;
}

Let’s now see what characters this Unicode range contains:

U+0000-00FF (I’m ommiting the control characters because they don’t render a character): 20!21"22#23$24%25&26'27(28)29*2A+2B,2C-2D.2E/2F030131232333434535636737838939:3A;3B<3C=3D>3E?3F@40A41B42C43D44E45F46G47H48I49J4AK4BL4CM4DN4EO4FP50Q51R52S53T54U55V56W57X58Y59Z5A[5B\5C]5D^5E_5F`60a61b62c63d64e65f66g67h68i69j6Ak6Bl6Cm6Dn6Eo6Fp70q71r72s73t74u75v76w77x78y79z7A{7B|7C}7D~7E A0¡A1¢A2£A3¤A4¥A5¦A6§A7¨A8©A9ªAA«AB¬ACAD®AE¯AF°B0±B1²B2³B3´B4µB5¶B6·B7¸B8¹B9ºBA»BB¼BC½BD¾BE¿BFÀC0ÁC1ÂC2ÃC3ÄC4ÅC5ÆC6ÇC7ÈC8ÉC9ÊCAËCBÌCCÍCDÎCEÏCFÐD0ÑD1ÒD2ÓD3ÔD4ÕD5ÖD6×D7ØD8ÙD9ÚDAÛDBÜDCÝDDÞDEßDFàE0áE1âE2ãE3äE4åE5æE6çE7èE8éE9êEAëEBìECíEDîEEïEFðF0ñF1òF2óF3ôF4õF5öF6÷F7øF8ùF9úFAûFBüFCýFDþFEÿFF
U+0131: ı131
U+0152-0153: Œ152œ153
U+02BB-02BC: ʻ2BBʼ2BC
U+02C6: ˆ2C6
U+02DA: ˚2DA
U+02DC: ˜2DC
U+2000-206F: 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 200A200B‌200C‍200D‎200E‏200F‐2010‑2011‒2012–2013—2014―2015‖2016‗2017‘2018’2019‚201A‛201B“201C”201D„201E‟201F†2020‡2021•2022‣2023․2024‥2025…2026‧2027 2028 2029‪202A‫202B‬202C‭202D‮202E 202F‰2030‱2031′2032″2033‴2034‵2035‶2036‷2037‸2038‹2039›203A※203B‼203C‽203D‾203E‿203F⁀2040⁁2041⁂2042⁃2043⁄2044⁅2045⁆2046⁇2047⁈2048⁉2049⁊204A⁋204B⁌204C⁍204D⁎204E⁏204F⁐2050⁑2051⁒2052⁓2053⁔2054⁕2055⁖2056⁗2057⁘2058⁙2059⁚205A⁛205B⁜205C⁝205D⁞205E 205F⁠2060⁡2061⁢2062⁣2063⁤2064⁥2065⁦2066⁧2067⁨2068⁩2069⁪206A⁫206B⁬206C⁭206D⁮206E⁯206F
U+2074: ⁴2074
U+20AC: €20AC
U+2122: ™2122
U+2191: ↑2191
U+2193: ↓2193
U+2212: −2212
U+2215: ∕2215
U+FEFF: FEFF
U+FFFD: �FFFD

As you can see, the Latin subset from Google Fonts contains a lot of characters, that your font may not even support. Furthermore, there’s a chance that you don’t need all these characters, or that you need some characters not listed above. In the following examples, you’ll see how to address these issues by creating your own subsets.

Examples

It’s time to use the pyftsubset to create some subsets. This is the list with the subsets you’ll create:

Keep all OpenType features and characters
Latin subset
English subset
Minimal English subset
Multiple subsets by script

Before we begin, I want to point out that I'm running the examples on a Windows machine with the Git Bash command line.

Keep all OpenType features and characters

The first thing you’ll do is to convert a TrueType (.ttf) file to a compressed TrueType file that’s optimized for the web (.woff2). You’ll keep all the characters and OpenType features from the original file. The file you’re working with is the regular version of Work Sans because it has many OpenType features and characters. Download the font file, open your command line in the downloads folder, and paste the following command:

pyftsubset WorkSans-Regular.ttf \
           --output-file="WorkSans-Regular-all.woff2" \
           --flavor=woff2 \
           --layout-features=* \
           --unicodes=*

The first parameter is the original font name (WorkSans-Regular.ttf). With the output-file option, you pass the name of the file you’ll create and with the flavor option the format of that file. You specify what OpenType features you want with the layout-features option—the asterisk means that you want them all. The unicodes option is for the Unicode characters you want, with the asterisk meaning that you want all the characters from the original file. If you specify more characters than the font file has, the program doesn’t throw an error by default. Instead of passing Unicode character codes, you can directly pass characters with the text or text-file options. I don’t prefer the text option because it seems error-prone.

I’m using the backslash (\) to span the command over multiple lines and make it more readable. The alternative is to remove the backslashes and write the command in a single line.

I suggest opening a text editor to paste the examples there so you can easily change them if you want. For example, the command above doesn’t work on the Windows CMD. To run multiline commands on the Windows command line, you’ll have to replace the backslash with a circumflex accent (`^`, `Shift + 6` in your keyboard).

You can also use the --verbose option to view information while the tool creates the file. With pyftsubset --help, you get a list of all the available options. Alternatively, check the pyftsubset documentation page that explains all the options.

The new font file is around 62kb and the original was 207kb. Along with the characters and the OpenType features, the font files also include hinting tables. These are instructions on how to render the fonts well on screens with small resolution (or more specifically, with low pixels per inch or ppi). If you want to remove the hinting, use the --no-hinting option along with the --desubroutinize option as shown in the following command:

pyftsubset WorkSans-Regular.ttf \
           --output-file="WorkSans-Regular-all-no-hinting.woff2" \
           --flavor=woff2 \
           --layout-features=* \
           --unicodes=* \
           --no-hinting \
           --desubroutinize

After removing the hinting, the file size goes down to 47kb, which means that hinting was taking up 30% of the file size. While testing hinting in a Windows machine, with a screen that has a 23″ diagonal and 1920 × 1080 resolution, I was able to see some minor differences between the hinted and the unhinted version of Work Sans. In mobile phones (they have high ppi) and Macs it shouldn’t matter much. As a result, I suggest to keep hinting, unless you really want to save some space.

Latin subset

Now, you’ll create a subset based on the Google font Latin subset you saw earlier. This subset includes a ton of useful characters, so it’s a good starting point if you want to cover text in English, French, Spanish, or German. This is the command you’ll run:

pyftsubset \
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-google-fonts.woff2" \
  --flavor=woff2 \
  --layout-features=* \
  --unicodes="U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,\
  U+02C6,U+02DA,U+02DC,U+2000-206F,U+2074,U+20AC,\
  U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD"

The new file has a file size of 35kb with hinting and 26kb without hinting. For comparison, the Latin Google font subset is around 23kb with hinting but with a limited number of layout features. To reduce the file size, you can limit the layout features too:

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-google-fonts-2.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,\
  U+02C6,U+02DA,U+02DC,U+2000-206F,U+2074,U+20AC,\
  U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD"

Let’s explain what the layout features do first (you already saw most of them):

In the first line, kern, liga, calt, clig are important features and they are on by default in CSS. ccmp, locl, mark, and mkmk are also always turned on by the browser, but their difference is that you can’t turn them off with CSS.
In the second line, we have some essential features for body text like the onum, pnum, smcp, and c2sc that you saw in a previous section, and for representing numbers we have frac, lnum, tnum, sups, and subs.
In the third line, we have some decoration features like cswh, dlig, ss01, ss03, and zero. If you don’t plan to use these features, you can remove them because some take a lot of space (cswh).

By including only these features, we save 4kb and we go down to 31kb. We were not able to reduce the size much because small caps and contextual swashes add a lot of extra glyphs to the file.

If you open the new file in wakamaifondue, you’ll notice that only the locl feature is kept from the features that you can’t change in CSS, and mark, ccmp, mkmk were thrown away. You can find an explanation for that in wakamaifondue:

These are the required layout features: features that are always turned on by the system that renders the font. You can’t turn them off in CSS. They may be applicable only to certain language scripts or specific languages, or in certain writing modes.

Because this process can get messy, If you don’t want to remove essential features from the original file by specifying the layout features explicitly with the = operator, you can instead add to the default features with += or remove with -=. The default features are calt, ccmp, clig, curs, dnom, frac, kern, liga, locl, mark, mkmk, numr, rclt, rlig, rvrn, and all features required for script shaping. Type pyftsubset --layout-features=? to see all the defaults features. So a safer alternative to the previous command is the following:

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-google-fonts-2.woff2" \
  --flavor=woff2 \
  --layout-features+="onum,pnum,smcp,c2sc,lnum,tnum,\
  subs,sups,cswh,dlig,ss01,ss03,zero"\
  --layout-features-="dnom,numr"\
  --unicodes="U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,\
  U+02C6,U+02DA,U+02DC,U+2000-206F,U+2074,U+20AC,\
  U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD"

Looking at the character table of the original file in wakamaifondue, I can see some cool characters we don’t have in the subset. I want to add all the arrows U+2190-21BB (← ↑ → ↓ ↔ ↕ ↖ ↗ ↘ ↙ ↩ ↪ ↺ ↻) and a hedgehog symbol () that’s on U+F8FF. To add those characters, change the unicodes option to the following:

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-google-fonts-3.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,\
  U+02C6,U+02DA,U+02DC,U+2000-206F,U+2074,U+20AC,\
  U+2122,U+2190-21BB,U+2212,U+2215,U+F8FF,U+FEFF,U+FFFD"

This increases the file size to 32kb but you now have some cool characters to work with.

You saw how to create a Latin subset with fewer characters and OpenType features. In the next section, you’ll remove even more characters to create an English-only subset.

English subset

Let’s now assume that you display only English on your site. If that’s the case, you’ll want to remove the characters used by other languages. Don’t overdo it though because you may display names or places from other languages. In the next section, you’ll verify what characters you use with glyphhanger so stay tuned.

We want to remove a big chunk from the Latin-1 supplement (U+0080-00FF), but we want to keep punctuation characters, symbols, and a couple of math characters: U+00D7 (×) and U+00F7 (÷). Long story short, from the following table, you’ll remove the highlighted characters:

20!21"22#23$24%25&26'27(28)29*2A+2B,2C-2D.2E/2F030131232333434535636737838939:3A;3B<3C=3D>3E?3F@40A41B42C43D44E45F46G47H48I49J4AK4BL4CM4DN4EO4FP50Q51R52S53T54U55V56W57X58Y59Z5A[5B\5C]5D^5E_5F`60a61b62c63d64e65f66g67h68i69j6Ak6Bl6Cm6Dn6Eo6Fp70q71r72s73t74u75v76w77x78y79z7A{7B|7C}7D~7E A0¡A1¢A2£A3¤A4¥A5¦A6§A7¨A8©A9ªAA«AB¬ACAD®AE¯AF°B0±B1²B2³B3´B4µB5¶B6·B7¸B8¹B9ºBA»BB¼BC½BD¾BE¿BFÀC0ÁC1ÂC2ÃC3ÄC4ÅC5ÆC6ÇC7ÈC8ÉC9ÊCAËCBÌCCÍCDÎCEÏCFÐD0ÑD1ÒD2ÓD3ÔD4ÕD5ÖD6×D7ØD8ÙD9ÚDAÛDBÜDCÝDDÞDEßDFàE0áE1âE2ãE3äE4åE5æE6çE7èE8éE9êEAëEBìECíEDîEEïEFðF0ñF1òF2óF3ôF4õF5öF6÷F7øF8ùF9úFAûFBüFCýFDþFEÿFF

Outside of the Latin-1 supplement, you’ll also remove U+0131 (ı), U+0152-0153 (Œ œ), and the spacing modifier letters U+02B0-02FF.

This is the command that creates an English-only subset:

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-english.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+0000-00A0,U+00A2-00A9,U+00AC-00AE,U+00B0-00B7,\
  U+00B9-00BA,U+00BC-00BE,U+00D7,U+00F7,U+2000-206F,U+2074,U+20AC,\
  U+2122,U+2190-21BB,U+2212,U+2215,U+F8FF,U+FEFF,U+FFFD"

With those changes, the file size goes from 31kb to 27kb. In the next section, you see how to create a performance focused subset.

Minimal English subset

The goal here is to create a tiny subset that you can use for two-stage font loading. As a result, you’ll want to keep the file size as small as possible—under 10kb would be ideal. The English subset in the previous section won’t do because it still has a lot of characters you probably won’t use. Instead, I suggest to install glyphhanger and use it to crawl your website to see what characters you use. If you want something more generic, but with fewer characters than in the previous section, crawl a popular English website with good typography.

First, you want to install glyphhanger globally:

npm install -g glyphhanger

You can also install it as a dev dependency in your project with yarn add -D glyphhanger and run it from there with yarn glyphhanger.

Then, crawl your development server (or your production site), after adding the correct port:

glyphhanger http://localhost:3000 --spider --spider-limit=5

I will now crawl my website to see what characters I’m using:

glyphhanger https://markoskon.com --spider --spider-limit=5

These are the Unicode ranges I get back after running the command above:

U+A,U+20,U+22,U+25-29,U+2B-3E,U+41-59,U+61-7A,U+F3,
U+3A4,U+2014,U+2019,U+201C,U+201D,U+2020,U+2021,
U+2026,U+20AC,U+2190,U+F8FF,U+1F426,U+1F4D7,U+1F4F0,U+1F525

To be sure you get all the characters back, you can crawl all your pages with --spider-limit=0, but this will take some time.
You can get the characters a specific font uses with the family option (e.g. --family="monospace") or specify multiple families with --family="Georgia,Consolas".

I will now pass the string I got in the --unicodes option (after removing the 4 emojis at the end) to create an English subset:

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-glyphhanger.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+A,U+20,U+22,U+25-29,U+2B-3E,U+41-59,U+61-7A,U+F3,\
U+3A4,U+2014,U+2019,U+201C,U+201D,U+2020,U+2021,\
U+2026,U+20AC,U+2190"

This results in a font file at around 20kb. This is still way over the 10kb goal. Let’s remove the hinting first because, in this case, size is more important:

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-minimal.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+A,U+20,U+22,U+25-29,U+2B-3E,U+41-59,U+61-7A,U+F3,\
U+3A4,U+2014,U+2019,U+201C,U+201D,U+2020,U+2021,\
U+2026,U+20AC,U+2190"\
  --no-hinting\
  --desubroutinize

The font file without hinting is around 14kb which is still not good enough.

The goal of two-stage font loading is to reduce or eliminate the time the browser displays the invisible or the fallback font. To be more precise, we have 3 fonts: the fallback (e.g. sans-serif), the small fake (e.g. Work Sans Minimal), and the real font (e.g. Work Sans). We also have 2 transitions: from the fallback to the fake and from the fake to the real font. Here, we’re not concerned with the first transition, so we’ll focus on the second. In this transition, you’ll want to reduce the moving of the text, and, to do that, you’ll want to keep all the layout features that affect spacing. All the features we have here affect spacing in some way, so if you plan to use them all, you can’t reduce the size any further.

But there’s a good chance that you won’t use all of them, or that you’ll use them sparingly, and, as a result, they won’t cause a lot of text reflow. So if you keep only the default features (minus the swashes) along with lnum, tnum (which are the default number forms):

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-minimal-super-minimal.woff2" \
  --flavor=woff2 \
  --layout-features+="lnum,tnum"\
  --layout-features-="cswh"\
  --unicodes="U+A,U+20,U+22,U+25-29,U+2B-3E,U+41-59,U+61-7A,U+F3,\
U+3A4,U+2014,U+2019,U+201C,U+201D,U+2020,U+2021,\
U+2026,U+20AC,U+2190"\
  --no-hinting\
  --desubroutinize

The command above creates a file that’s 8kb in size which is under the 10kb goal.

Extra: Find what characters a language uses

This is what you can do when you want to cover a language, but you are not sure what characters to add in your subsets.

There is the Unicode Common Locale Data Repository (CLDR) that shows, among other things, what characters a language uses. For example, these are the characters for the French language. I also made an app a while ago that shows that data in a table.
You can crawl a popular website that uses that language with glyphhanger, as you saw in this section.
After you create your subsets, you can see if you cover all the characters with the Firefox developer tools. Open the dev tools, go to the inspector tab, and, at the top right, select the Fonts tab that’s sitting next to Layout/Computed/Changes.

Multiple subsets by script

A repo with the examples of this section

Instead of creating a single subset and throwing away the rest of the characters, you can keep them all by chunking the initial file into multiple files. You can then take advantage of the unicode-range descriptor—that you can use inside your @font-face—to instruct the browser to download only the chunk it needs to render the text. The easiest way to chunk the file is follow the Unicode blocks; for example, you can create the following subsets:

Set	Range
Latin	`U+0000-00FF`
Latin-extended-a	`U+0100-017F`
Latin-extended-b	`U+0180-024F`
Rest middle	`U+0259-03C0`
Latin-extended-additional	`U+1E00-1EFF`
Rest	`U+2000-FB02`

In this case, the pyftsubset commands will be:

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-latin.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+0000-00FF" &&

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-latin-extended-a.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+0100-017F" &&

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-latin-extended-b.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+0180-024F" &&

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-rest-middle.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+0259-03C0" &&

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-latin-extended-additional.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+1E00-1EFF" &&

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-rest.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+2000-FB02"

And this is the CSS code that uses the files, assuming you serve them from a fonts folder:

/* Regular */
@font-face {
  font-family: "Work Sans";
  font-display: swap;
  font-weight: 400;
  font-style: normal;
  src: url("/fonts/WorkSans-Regular-latin.woff2") format("woff2");
  unicode-range: U+0000-00FF;
}
@font-face {
  font-family: "Work Sans";
  font-display: swap;
  font-weight: 400;
  font-style: normal;
  src: url("/fonts/WorkSans-Regular-latin-extended-a.woff2") format("woff2");
  unicode-range: U+0100-017F;
}
@font-face {
  font-family: "Work Sans";
  font-display: swap;
  font-weight: 400;
  font-style: normal;
  src: url("/fonts/WorkSans-Regular-latin-extended-b.woff2") format("woff2");
  unicode-range: U+0180-024F;
}
@font-face {
  font-family: "Work Sans";
  font-display: swap;
  font-weight: 400;
  font-style: normal;
  src: url("/fonts/WorkSans-Regular-rest-middle.woff2") format("woff2");
  unicode-range: U+0259-03C0;
}
@font-face {
  font-family: "Work Sans";
  font-display: swap;
  font-weight: 400;
  font-style: normal;
  src: url("/fonts/WorkSans-Regular-latin-extended-additional.woff2")
    format("woff2");
  unicode-range: U+1E00-1EFF;
}
@font-face {
  font-family: "Work Sans";
  font-display: swap;
  font-weight: 400;
  font-style: normal;
  src: url("/fonts/WorkSans-Regular-rest.woff2") format("woff2");
  unicode-range: U+2000-FB02;
}

You’ll want to do the same thing for the rest of the weights you want to support and for the italics. This method doesn’t work on Internet Explorer because it downloads all the files. The site is still usable but it’s kind of a disaster performance-wise.

The following table shows the sizes of the subsets after running the commands:

Filename	Size
WorkSans-Regular-latin.woff2	29kb
WorkSans-Regular-latin-extended-a.woff2	15kb
WorkSans-Regular-latin-extended-b.woff2	10kb
WorkSans-Regular-rest-middle.woff2	5kb
WorkSans-Regular-latin-extended-additional.woff2	13kb
WorkSans-Regular-rest.woff2	12kb

This means that, for English content, the browser will download the Latin version (29kb) and maybe the rest (12kb) which is 41kb in total for the regular normal weight. If you have italics and a bold version, the browser will download 123kb (41 × 3) in total, which is not ideal.

A different way to chunk the initial file is to use the subset you created earlier—that has the English characters, the punctuation, and the symbols—and create subsets for the rest blocks. See the following list for an example:

English/punctuation/symbols: U+0000-00A0, U+00A2-00A9, U+00AC-00AE, U+00B0-00B7, U+00B9-00BA, U+00BC-00BE, U+00D7, U+00F7, U+2000-206F, U+2074, U+20AC, U+2122, U+2190-21BB, U+2212, U+2215, U+F8FF, U+FEFF, U+FFFD
French/German/etc.: U+00A1, U+00AA-00AB, U+00AF, U+00B8, U+00BB, U+00BF-00D6, U+00D8-00F6, U+00F8-00FF, U+0131, U+0152-0153, U+02B0-02FF
Latin-extended-a: U+0100-0130, U+0132-0151, U+0154-017F
Latin-extended-b: U+0180-024F
Latin-extended-additional: U+1E00-1EFF
The rest: U+0259, U+0300-03C0, U+2070-2073, U+2075-20AB, U+20AD-2121, U+2123-218F, U+21BC-2211, U+2213-2214, U+2216-F8FE, U+FB01-FB02

These are the new commands:

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-english.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+0000-00A0,U+00A2-00A9,U+00AC-00AE,U+00B0-00B7,\
  U+00B9-00BA,U+00BC-00BE,U+00D7,U+00F7,U+2000-206F,U+2074,U+20AC,\
  U+2122,U+2190-21BB,U+2212,U+2215,U+F8FF,U+FEFF,U+FFFD" &&

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-rest-latin.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+00A1,U+00AA-00AB,U+00AF,U+00B8,U+00BB,U+00BF-00D6,\
  U+00D8-00F6,U+00F8-00FF,U+0131,U+0152-0153,U+02B0-02FF" &&

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-latin-extended-a.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+0100-0130,U+0132-0151,U+0154-017F" &&

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-latin-extended-b.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+0180-024F" &&

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-latin-extended-additional.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+1E00-1EFF" &&

pyftsubset\
  WorkSans-Regular.ttf \
  --output-file="WorkSans-Regular-rest.woff2" \
  --flavor=woff2 \
  --layout-features="kern,liga,clig,calt,ccmp,locl,mark,mkmk,\
  onum,pnum,smcp,c2sc,frac,lnum,tnum,subs,sups,\
  cswh,dlig,ss01,ss03,zero"\
  --unicodes="U+0259,U+0300-03C0,U+2070-2073,U+2075-20AB,\
  U+20AD-2121,U+2123-218F,U+21BC-2211,U+2213-2214,U+2216-F8FE,\
  U+FB01-FB02"

And these are the file sizes:

Filename	Size
WorkSans-Regular-english.woff2	27kb
WorkSans-Regular-rest-latin.woff2	10kb
WorkSans-Regular-latin-extended-a.woff2	14kb
WorkSans-Regular-latin-extended-b.woff2	10kb
WorkSans-Regular-latin-extended-additional.woff2	13kb
WorkSans-Regular-rest.woff2	11kb

Now the browser will download 27kb for English content and 81kb in total for regular, italic, and bold.

A note for overlaps when using the unicode-range

Because the process of figuring out which characters go to which file is time-consuming and error-prone, you can use Unicode ranges that overlap. The following is a quote from the unicode-range descriptor in the CSS Font Module specification.

If the Unicode ranges overlap for a set of @font-face rules with the same family and style descriptor values, the rules are ordered in the reverse order they were defined; the last rule defined is the first to be checked for a given character.

In other words, if you have overlaps in the Unicode ranges, put the frequently used subset last, the one that is most likely to be downloaded (e.g. English/punctuation/symbols), and the rest of the files above it. So you can create a complex English subset, and for the rest of the characters, create a subset for each Unicode block:

/* Regular */
@font-face {
  font-family: "Work Sans";
  font-display: swap;
  font-weight: 400;
  font-style: normal;
  src: url("/fonts/WorkSans-Regular-latin.woff2") format("woff2");
  unicode-range: U+0000-00FF;
}
@font-face {
  /* … */
  src: url("/fonts/WorkSans-Regular-latin-extended-a.woff2") format("woff2");
  unicode-range: U+0100-017F;
}
@font-face {
  /* … */
  src: url("/fonts/WorkSans-Regular-latin-extended-b.woff2") format("woff2");
  unicode-range: U+0180-024F;
}
@font-face {
  /* … */
  src: url("/fonts/WorkSans-Regular-latin-extended-additional.woff2")
    format("woff2");
  unicode-range: U+1E00-1EFF;
}
@font-face {
  /* … */
  src: url("/fonts/WorkSans-Regular-rest.woff2") format("woff2");
  unicode-range: U+0259-03C0, U+2000-FB02;
}
/* Place the English subset last */
@font-face {
  font-family: "Work Sans";
  font-display: swap;
  font-weight: 400;
  font-style: normal;
  src: url("/fonts/WorkSans-Regular-english.woff2") format("woff2");
  unicode-range: U+0000-00A0, U+00A2-00A9, U+00AC-00AE, U+00B0-00B7,
    U+00B9-00BA, U+00BC-00BE, U+00D7, U+00F7, U+2000-206F, U+2074,
    U+20AC, U+2122, U+2190-21BB, U+2212, U+2215, U+F8FF, U+FEFF, U+FFFD;
}

Raph Levien and Jason Pamental made a really good note in the 2020 Web almanac about fonts that if you organize your subsets by Unicode blocks, you may end up dropping kerning pairs. For example, if you create a subset for the basic Latin (U+00-FF), the kerning between letter W and À (WÀ) will be fine. But if you create one for basic Latin (U+00-7F W lives here) and one for Latin-1 supplement (U+80-FF À lives here), you will drop that kerning pair. This is probably true for more OpenType features.

Recap

Let’s now make a quick recap of the key points of this post:

You saw why you may want to subset your fonts.
Characters and OpenType features are two of the most important things font files include. You also saw what some OpenType features do.
wakamaifondue is an online tool that can help you analyze font files. You used it to see what the Google font files contain.
You saw how to install and use pyftsubset to create custom font subsets. More specifically:
- How to create a web font file with all the characters and features from the original.
- How to create a Latin subset based on the Google fonts character list but with more layout features.
- How to create an English only subset.
- How to use glyphhanger to create a tiny subset for two-stage font loading.
- Finally, how to keep all the characters by breaking up the initial file into chunks based on the Unicode block they belong to.

Feedback is welcome. This includes corrections, suggestions, and things to explain further.

Appendix

This section contains information that’s useful when working with subsets. You definitely don’t have to read it.

Conversions in JavaScript
Manual conversions
Unicode ranges to characters
Unicode blocks & explanations
Other Notes
Links

Conversions in JavaScript

See the character behind a Unicode hex number.
1. Convert a hex string to a decimal (base-ten) number:
```
Number.parseInt("A9", 16); // 169
```
2. Convert that decimal number (or that code point) to a character:
```
String.fromCharCode(169); // prints the copyright symbol: ©
// or fromCodePoint()
```
Get the Unicode hex number for a character
1. And the other way around, get the char code for the copyright symbol:
```
"©".charCodeAt(); // 169
// or codePointAt()
```
2. Convert the char code to a hex string:
```
Number(169).toString(16); // A9 or U+00A9
```

Manual conversions

Convert a hex number to a decimal with pen and paper. The available digits are from 0 to F; A is equal to 10; F is equal to 15.

FF = 15×16¹ + 15×16⁰
   = 240    + 15
   = 255

BB8 = 11×16² + 11×16¹ + 8×16⁰
    = 2816   + 176    + 8
    = 3000

FFEF = 15×16³ + 15×16² + 14×16¹ + 15×16⁰
     = 61440  + 3840   + 224    + 15
     = 65519

Unicode ranges to characters

JavaScript code that takes as input a bunch of Unicode ranges and outputs the characters. 100% not tested.

`U+20,U+27-29,U+2C-2E,U+30-3B,U+41-49,U+4B-50,U+52-56,U+59,
U+5A,U+61-70,U+72-7A,U+7C,U+A0,U+A9,U+107,U+10C,U+10D,U+111,
U+161,U+17E`
  .replace(/U\+/g, "")
  .split(",")
  .reduce((res, item) => {
    if (/-/.test(item)) {
      const [first, last] = item.split("-").map((i) => parseInt(i, 16));
      let numbers = [];
      for (let i = first; i <= last; i++) {
        numbers.push(String.fromCharCode(i));
      }
      return [...res, { range: item, characters: numbers.join(" ") }];
    } else {
      return [
        ...res,
        {
          range: item,
          characters: String.fromCharCode(parseInt(item, 16)),
        },
      ];
    }
  }, []);

Create a Unicode table

Display the Unicode Basic Multilingual Plane (BMP) characters (U+0000-FFFF) in a few lines of code:

// FFFF is 65535 in decimal
new Array(65536)
  .fill(1)
  .map((one, index) => one * index)
  .map((codePoint) => String.fromCodePoint(codePoint));

// or with code points embeded:
new Array(65536)
  .fill(1)
  .map((one, index) => one * index)
  .map((codePoint) => ({
    character: String.fromCodePoint(codePoint),
    codePoint,
    unicode: Number(codePoint)
      .toString(16)
      .padStart(6, "U+000")
      .toUpperCase(),
  }));

Unicode blocks & explanations

See also the Unicode Names List Charts. It presents the Unicode blocks in tables with information for each character.

U+0000-00FF (256 characters). Includes control characters (U+0000-001F, 32 characters), basic Latin (U+0020-007F, 96 characters; includes English and general punctuation), the Latin-1 supplement (U+0080-00FF, 128 characters for French, Italian, Spanish, German, and Icelandic).
U+0100-017F (Latin Extended-A, 128 characters). Supports Maltese, Polish, Czech, Hungarian, Serbo-Croatian, Turkish, and more.
U+0180-024F (Latin Extended-B, 208 characters). Supports African, historic, additions for Romanian, Croatian, Slovenian.
U+0250-02FF = 174: Phonetic stuff, and more specifically: IPA Extensions (U+0250-02AF = 96) and spacing modifier letters (U+02B0-02FF = 80). Some languages use characters from these sets.
U+0300-036F = 112. Combining diacritical marks. Some languages use characters from this set—see the Unicode PDF. As I understand this, you use them to type text in many languages, including polytonic Greek. When you render text, though, these characters merge to form characters from the script Unicode blocks.
U+0370-03FF (Greek and Coptic = 144, neohellenic monotonic)
U+0400-04FF (Cyrillic = 256).
U+0500-052F (Cyrillic Supplement = 48).
U+1D00-1D7F (Phonetic Extensions = 128).
U+1D80-1DBF (Phonetic Extensions Supplement = 64).
U+1DC0-01DFF (Combing Diacritical Marks Supplement = 64).
U+1E00-1EFF (Latin Extended Additional = 256).
U+1F00-1FFF (Greek extended = 256, polytonic Greek)
U2000-FEFF Rest goodies. This includes:
- general punctuation (U+2000-206F).
- superscripts/subscripts (U+2070-209F).
- currency symbols (U+20A0-20CF, or just the euro 20AC).
- Letter-like symbols (U+2100-214F, includes TM, Tel. No symbols).
- Arrows (U+2190-21FF),
- Mathematical operators (U+2200-22FF, minus, division signs)
- and many more.

Other Notes

Unicode property escapes in JavaScript regular expressions seem very useful, and more specifically: General_Category, Script, and Script_Extensions.
There are standardized subsets for Unicode but they are big.
There are other Latin sets later in the Unicode table, such as the Latin Extended Additional, and the Latin Extended-C/D/E.
You can find punctuation in ASCII, Latin-1 supplement, general punctuation, and supplemental punctuation (U+2E00-2E7F).
The “ugly” quote lives in ASCII, the triangle (Italian) in Latin-1 supplement, and the curly quotes in general punctuation.
Unicode has small caps in IPA Extensions, Phonetic Extensions, and Latin Extended-D, but they are meant for use in the phonetic alphabet (?). Use OpenType small caps instead.
Some typefaces have superscripts and subscripts in Unicode while others have them in OpenType features. Prefer OpenType because they may also include letters. Subscripts and superscripts in Wikipedia.
There are blocks for ancient Greek numbers, linear A, and linear B.

Reforma typeface notes

Fri, 20 Mar 2020 19:00:00 GMT

You can find here my notes for the Reforma typeface. Reforma was designed by Alejandro Lo Celso and programmed by Guido Ferreyra as part of a commission to the PampaType foundry by the Universidad Nacional de Córdoba, Argentina. It is published under the Creative Commons CC BY-ND 4.0 and you can use it for free with attribution. It consists of 3 subfamilies: the Reforma 1918, the Reforma 1969, and the Reforma 2018.

Opinions & notes

The following quotes are from a blog post written by the designer that explains the story of the typeface, and from the PDF files that come with the font files.

"The Reforma type family is divided into 3 subfamilies:
Reforma 1918, the classic version (humanist serif).
Reforma 2018, the modern version (humanist sans-serif).
Reforma 1969, a hybrid version (incise, flare serif)".

Reforma is a multi-form humanist typeface that combines the virtues of Roman epigraphic tradition, such as dignity and sobriety, with a sense of modern friendliness.

The following quote is from a Typographica review by Luara Meseguer.

…Reforma explores versatility by way of a common skeleton wearing different outfits.

My notes

Reforma 1918 looks good in large sizes, but I prefer the other two for body text.
Reforma 2018 (the sans-serif) version gives me a “calculator vibe”. This happens due to the extreme modulated strokes of the bold versions, near the joining strokes. For example:
n h p
When you download the font files, you get a specimen PDF and a PDF with usage instructions.
It doesn’t have a regular weight, but you can use the Gris version (500) with lighter colors.
The zip includes woff/woff2 files, and because they are small, there’s no need for Unicode subsets.
It has a black 900 version, but it’s used exclusively by the university.

Playground

All notes are for the following version:

Version 1.001; ttfautohint (v1.6)

Sub-family: Weight: Style: Width: Size: 20pxLine height: 1.50emLetter spacing: 0.00em

A peep at some distant orb has power to raise and purify our thoughts like a strain of sacred music, or a noble picture, or a passage from the grander poets. It always does one good.

OpenType features

All the Reforma subfamilies support 22 layout features in roman and 25 in italics. They support all types of figures (onum, pnum, tnum, lnum), but they doesn’t have small caps. Reforma has many interesting stylistic sets, and the italic versions have discretionary ligatures. There are also some uppercase ligatures but they are exclusively used by the University. The following table shows some of the most interesting features:

Sub-family: Weight:

Feature code	On	Off	Notes
`salt or ss01`	J K R Q	J K R Q	All stylistic alternates.
`ss02`	Jupiter	Jupiter	Capital J without a descender.
`ss03`	ROCK	ROCK	Long leg in capital K & R.
`ss04`	Quality	Quality	Capital Q with a long tail.
`ss05 (italic)`	Quality	Quality	Italic capital Q with an extra-long tail.
`ss06`	Tome	Tome	Taller capital T.
`ss07 (italic)`	Ysera	Ysera	Alternate italic capital Y.
`dlig (italic)`	effect station	effect station	Italic discretionary ligatures.
`ss05 + dlig + italic`	Question	Question	Combination.

Interesting characters

For more interesting characters see the OpenType features above, check the playground, or get a copy from Pampatype’s site and drop it in the Wakamai Fondue app to discover them yourself.

C
H
J
M
Q
R
a
f
n
r
9
&
€
“”
†
‡


Character support

According to Wakamai Fondue, it has 276 characters and 359 glyphs. The font files cover the majority, if not all, of the following Unicode sets.

Set	Range
Latin	`U+0000-00FF`
Rest	`U+2000-FB01`

Not complete sets (below 50%)

Total characters = 42. This means that the typeface supports only specific languages/characters from these Unicode blocks.

Set	Range	Notes
Latin Extended-A	`U+0100-017F`	14 chars
Latin Extended-B	`U+0180-024F`	1 char
Spacing Modifier Letters	`U+02B0-02FF`	5 chars
Combing Diacritical Marks	`U+0300-036F`	11 chars
Greek	`U+0370-03FF`	3 chars
Latin Extended Additional	`U+1E00-1EFF`	8 chars

Weights & styles

Each subfamily comes in 3 weights with matching italics: light 300 (Blanca), semi 500 (Gris), and bold 700 (Negra). Italic = Italica.

Using the contenteditable attribute

Tue, 17 Mar 2020 18:00:00 GMT

Instead of using a <textarea>, you can make any HTML element editable by setting the contenteditable attribute to true—see the MDN link and the specification link for more details on the attribute. This way, you can have an element that behaves like a textarea but it automatically resizes when the user types—something not possible with a textarea. This behavior can be useful for font-related applications, where you may not care what’s inside the editable element.

Uncontrolled component

If you’re using React, you can use a contenteditable element as an uncontrolled component. This means you don’t keep in the state the text that’s inside the editable element. The following snippet shows an example of that:

<p
  suppressContentEditableWarning
  contentEditable
  spellCheck={false}
>
  Edit this text
</p>

Instead of “Edit this text” you could render a children prop. This is what the code snippet above looks like when rendered in React:

Edit this text

If you don’t use the suppressContentEditableWarning, React will give you the following warning:

A component is contentEditable and contains children managed by React. It is now your responsibility to guarantee that none of those nodes are unexpectedly modified or duplicated. This is probably not intentional.

We are ok with the situation the warning describes because we don’t care what’s inside, so we suppress it.

If you want to have the resize handle at the bottom right of the element use the following CSS:

.editable {
  resize: both;
  overflow: auto;
}

The next element has a resize handle at the bottom right. It resizes when the user types until they drag the handle. At that point, it behaves like a textarea.

You can resize me

Pasting only text

If you try to paste something inside that element, you’ll notice that you can paste HTML elements, not only text. Try copying the “Pasting only text” heading into the previous editable element. If you want to paste only text, you can override the default behavior with an onPaste listener.

In the following snippet, I prevent the default element paste with e.preventDefault(), and I take only the text from the clipboard with the getData("text") method. Finally, I insert the text at the caret position with the document.execCommand:

import React from "react";

export default () => (
  <p
    suppressContentEditableWarning
    contentEditable
    spellCheck={false}
    onPaste={(e) => {
      e.preventDefault();
      const text = e.clipboardData.getData("text");
      document.execCommand("insertText", false, text);
    }}
  >
    Only text is allowed here.
  </p>
);

Try pasting a heading or a code block in the following paragraph. You’ll notice that you can paste only text.

Only text is allowed here.

The MDN link for execCommand warns us that this feature is now obsolete and could be removed at any time. You’d think that instead of using the execCommand, you can use a ref and directly change the DOM, as I do below:

import React, { useRef } from "react";

export default () => {
  const editableRef = useRef();
  return (
    <p
      suppressContentEditableWarning
      contentEditable
      ref={editableRef}
      spellCheck={false}
      onPaste={(e) => {
        e.preventDefault();
        const text = e.clipboardData.getData("text");
        editableRef.current.innerText = text;
      }}
    >
      Only text is allowed here.
    </p>
  );
};

But in this case, you replace the content every time, and, as a result, you lose the caret position because the content is brand new. This is what it looks like if you do that:

Probably not what you want.

Besides that, the undo command Ctrl + Z doesn’t work. If you want to implement the functionality of the insertText command, you have to re-invent the wheel—I have some code at the end with unsuccessful experiments. You have to account for the current selection to paste the text in the correct position, and after that, you have to place the caret at the expected position by creating a range. As a result, sticking with the obsolete functionality or using a <textarea> seems the way to go.

The support for e.clipboardData object doesn’t seem to be that great, at the time of writing at least. MDN warns us that it’s an experimental technology. I wonder if there is some error on caniuse.com, or if I can’t read the tables properly. For example, it says that’s not supported on desktop Safari 13, but I found that’s not the case when I tested. Anyway, an alternative is to use the Clipboard from the Navigator API which seems to have better support.

“Controlled” component

If you want to use the text of the editable element, you’ll have to keep it in the state. To update that state, you can use the onInput listener. But you’ll face the same problems I mentioned before with the onPaste listener. The problem is that you don’t have a value property for the editable element (as you have with inputs), nor an event.target.value on the event object of the onInput listener. If you replace the text content with a ref—mutate the DOM directly—you lose the caret position.

There is a small NPM package called react-contenteditable that approaches this problem in a different way. When the text content changes, you don’t replace the HTML content, but you call the user’s onChange listener and pass the current value so the user can update the state. You get that value with a ref, for example: ref.current.innerText. At the same time, you store that value in a variable outside the state. Because the user value (state) changes, the component will want to re-render. You prevent this with a shouldComponentUpdate method. This is the source code of the component on GitHub, and this an example on how to use it:

import React, { useState, useRef } from "react";
import ContentEditable from "react-contenteditable";

export default () => {
  const editableRef = useRef();
  const [editableText, setEditableText] = useState("Edit me.");
  return (
    <ContentEditable
      innerRef={editableRef}
      tagName="p"
      html={editableText}
      onPaste={(e) => {
        e.preventDefault();
        const text = e.clipboardData.getData("text");
        document.execCommand("insertText", false, text);
      }}
      onChange={(e) => {
        const html = e.target.value;
        setEditableText(html);
      }}
    />
  );
};

If you want to paste only text, you still have to implement the onPaste listener.

Final thoughts

Depending on your requirements, working with contentEditable elements in React can be challenging. Below, you can find more of my thoughts regarding the contentEditable attribute. If you feel I’m wrong somewhere or there is a better way of approaching the problem, let me know in the comments.

If your use case is to have text that when the user clicks inside, it becomes a contenteditable element, a better solution from an accessibility standpoint is to use a plain old input. Assuming you do this because you don't like the look of the input. If that's the case, you can use CSS to change the styling.

contenteditable elements can do much more; you can use them as rich text editors for example. Take a look at the MDN guide “Making Content Editable”.

You have to be aware of Cross-Site Scripting (XSS) attacks. If you allow the user to add HTML inside an element, and you then save that value in a server without sanitizing it, you may be vulnerable to XSS attacks. That is if you try to render again that content on another client. An example of that is comments from two different users on a post.

Stackoverflow thread from 2014; React is old I guess.
Which elements can be safely made contenteditable?
The autocapitalize attribute.
CSS Tricks: Show and Edit Style Element.
Tania Rascia: Using Content Editable Elements in JavaScript (React).

Implementing insertText (without success)

This is what I came up with while trying to implement the insertText command on the onPaste listener.

// onPaste listener
e.preventDefault();
const pastedText = e.clipboardData.getData("text");
const selection = window.getSelection();
const currentText = pRef.current.innerText;

// The idea was to create a range before
// I replace the text to keep track of
// the caret position. I add it to the
// selection aftewards. This doesn't work.
const range = document.createRange();
range.setStart(selection.anchorNode, selection.anchorOffset);
range.collapse(true);

// The selection is collapsed when the user
// has not selected any of the existing text.
// This works assuming the content is a single
// text node. If you press the enter to create
// new lines, that's not the case anymore and it
// doesn't work.
if (selection.isCollapsed) {
  const beforeCursor = currentText.slice(0, selection.anchorOffset);
  const afterCursor = currentText.slice(selection.anchorOffset);
  pRef.current.innerText = beforeCursor + pastedText + afterCursor;
} else {
  const selectionStart = currentText.slice(0, selection.anchorOffset);
  const selectionEnd = currentText.slice(selection.focusOffset);
  pRef.current.innerText = selectionStart + pastedText + selectionEnd;
}

selection.removeAllRanges();
selection.addRange(range);

Add inline style elements in Gatsby

Sun, 15 Mar 2020 11:00:00 GMT

I wanted to add inline style elements that are unique per page in this Gatsby blog. My use case was to create several posts where I would share my notes about some typefaces I like from Google Fonts. Inside those posts, I would download the font files, display an interactive component where the user can add text and change font-related CSS attributes, and, finally, show some interesting characters and OpenType features.

I didn’t want the font face rules to be present in all pages, something that would increase the HTML size. If you want that, you have a ton of options. Let’s name a few:

You can import a CSS file inside your gatsby-browser.js file.
You can import a CSS file inside any React component.
You can create a “global style” with a CSS-in-JS library like styled-components or emotion.
You can use the Gatsby SSR APIs. More specifically, the onRenderBody and its parameters setHeadComponents and setPostBodyComponents.

Coming back to the initial requirement, if you want to add an inline style element that would apply only to a specific page in Gatsby, you can do the following:

src/pages/page-2.js

const Page2 = () => (
  <Layout>
    <Helmet>
      <style type="text/css">
        {`
        @font-face {
          font-family: "Source Sans Variable";
          font-display: swap;
          font-weight: 200 900;
          font-style: normal;
          font-stretch: normal;
          src: url("/fonts/SourceSansVariable-Roman-latin.woff2") format("woff2");
          unicode-range: U+00-FF;
        }
        ... skipping the rest
      `}
      </style>
    </Helmet>
    <h1>Page 1</h1>
    {/* content */}
  </Layout>
);

In my case, the pages were not JSX files but MDX, and the above snippet didn’t work inside an MDX page. The workaround was simple—and maybe preferred from a “clean code” perspective. I created a react-helmet only component with the font faces, and then I imported that component inside the MDX. This the component I’m talking about:

content/posts/source-sans/font-faces.jsx

import React from "react";
import { Helmet } from "react-helmet";

export default () => (
  <Helmet>
    <style type="text/css">
      {`
        @font-face {
          ... skipping
        }
      `}
    </style>
  </Helmet>
);

And this is the import/render inside the MDX page:

content/posts/source-sans/index.mdx

import FontFaces from "./font-faces";

<FontFaces />;

The GitHub page for the react-helmet has a reference guide that shows how to add other types of elements in the head of the document.

Don't forget that to use react-helmet properly in Gatsby, you need the gatsby-plugin-react-helmet plugin.

GitHub issue: Using different CSS file for different pages.

Work Sans typeface notes

Thu, 12 Mar 2020 19:00:00 GMT

You can find here my notes for the Work Sans typeface. Work Sans is designed by Wei Huang and is published under the SIL Open Font License Version 1.1.

Opinions & notes

Description from the creator and some personal notes in the end. The following quotes are from the project’s site.

Work Sans is based loosely on early Grotesques — i.e. Stephenson Blake, Miller & Richard and Bauerschen Giesserei.

The core of the fonts are optimized for on-screen medium-sized text usage (14px-48px) – but still can be used in print well. The fonts at the extreme weights are designed more for display use.

My notes

It has a variable font.
It’s playful and has a ton of features for an open-source typeface.

Playground

I load the variable version of Work Sans. Below is the exact version of the variable fonts I use.

Version 2.007; ttfautohint (v1.8.1.43-b0c9)

Weight: Style: Width: Size: 20pxLine height: 1.50emLetter spacing: 0.00em

A peep at some distant orb has power to raise and purify our thoughts like a strain of sacred music, or a noble picture, or a passage from the grander poets. It always does one good.

OpenType features

Work Sans supports 39 layout features. It supports features essential for body text, such as old-style numbers (onum), proportional numbers (pnum), small caps, and it also has a ton of stylistic sets. It also has support for stylistic features such as swashes (swsh, cswh) and discretionary ligatures (dlig). The following table shows some of the most interesting to me:

Feature code	Result	Default	Notes
`swsh`	A E F L M N R S T	A E F L M N R S T	Capital swashes
`swsh`	n r t y	n r t y	Lowercase swashes
`cswh`	Reality	Reality	Contextual swashes compared to swashes
`dlig`	effect station	effect station	Discretionary ligatures
`cswh + dlig`	Rocksteady	Rocksteady	Contextual swashes and discretionary ligatures combined
`ornm`	• “ ” ‘ ’ ¶	• “ ” ‘ ’ ¶	Ornaments
`ss02`	l	l	Lowercase l without a tail.
`ss03`	R	R	Capital R alternate
`ss04`	G	G	Capital G alternate
`ss06`	a	a	Single story lowercase a

Interesting characters

For more interesting characters see the OpenType features above, check the playground, or get a copy from GitHub and drop it in the Wakamai Fondue app to discover them yourself.

Character support

According to Wakamai Fondue, it has 753 characters and 1410 glyphs. The font files cover the majority if not all of the following Unicode sets.

Set	Range
Latin	`U+0000-00FF`
Latin Extended-A	`U+0100-017F`
Latin Extended-B	`U+0180-024F`
Latin Extended Additional	`U+1E00-1EFF`
Rest	`U+2000-FB01`

Not complete sets (below 50%)

Total characters = 48. This means that the typeface supports only specific languages/characters from these Unicode blocks. See the project’s site for detailed language support.

Set	Range	Notes
IPA Extensions	`U+0250-02AF`	1 char
Spacing Modifier Letters	`U+02B0-02FF`	19 chars
Combing Diacritical Marks	`U+0300-036F`	25 chars
Greek	`U+0370-03FF`	3 chars

Weights & styles

It comes in 9 weights with matching italics. It’s also available in a variable font format with a weight variation axis that supports weights from 100 to 900.

Source Sans Pro typeface notes

Wed, 11 Mar 2020 19:00:00 GMT

You can find here my notes for the Source Sans Pro typeface. Source Sans was designed by Paul D. Hunt and is licensed under the SIL Open Font License, Version 1.1.

Opinions & notes

Opinions from the creators and other credible sources. In the end, there are some personal notes.

Typewolf

The design was inspired by American gothics, however, some traits such as the tail on the lowercase l are more reminiscent of humanist typefaces.

The slightly condensed letterforms of Source Sans Pro make it ideal for user interfaces where space is a concern.

A note from Captain obvious: it also has the modulated stroke of the humanist typefaces.

Adobe blog

Source Sans is a classic grotesque typeface with a simple, unassuming design.

In UI screenshots, Source Sans feels more classically typographic than I would expect.

Source Sans on Adobe blog.

My notes

Arguably the best and most complete family on Google fonts. Don’t forget that the Source family includes a serif (Source Serif Pro) and a monospace version (Source Code Pro). Additionally, there is an alternative for the monospace version called Hasklig that includes code ligatures.
It supports Greek and they are awesome.

Playground

All the notes and the interactive controls below are for the variable version of Source Sans Pro; you can see the exact version I used below.

Version 3.006; hotconv 1.0.111; makeotfexe 2.5.65597

Weight: Style: Width: Size: 20pxLine height: 1.50emLetter spacing: 0.00em

A peep at some distant orb has power to raise and purify our thoughts like a strain of sacred music, or a noble picture, or a passage from the grander poets. It always does one good.

OpenType features:

Source Sans Pro supports 52 layout features. It supports features essential for body text, such as old-style numbers (onum), proportional numbers (pnum), small caps, and it also has a ton of stylistic sets and character variants. The following table shows some of the most interesting to me:

Feature code	Result	Default	Notes
`ss04 or cv03`	g	g	Single story lower case g
`ss05 or cv04`	l	l	Lower case l without a tail.
`ss06 or cv05`	a	a	Double story lower case a with a tail
`ss02`	a g l	a g l	-

Interesting characters

Many stylistic sets and characters variants. It has a ton of characters U+2000+. For more interesting characters, check the playground or get a copy from GitHub and drop it in the Wakamai Fondue app.

M
Q
S
a
g
l
t
💩
🤖

Character support

According to Wakamai Fondue, it has 1436 characters and 2194 glyphs. The font files cover the majority if not all of the following Unicode sets.

Set	Range	Notes
Latin	`U+0000-00FF`
Latin Extended-A	`U+0100-017F`
Latin Extended-B	`U+0180-024F`	probably below 50%
IPA Extensions	`U+0250-02AF`
Spacing Modifier Letters	`U+02B0-02FF`
Combing Diacritical Marks	`U+0300-036F`
Greek	`U+0370-03FF`
Cyrillic	`U+0400-04FF`
Latin Extended Additional	`U+1E00-1EFF`
Greek Extended	`U+1F00-1FFF`
Rest	`U+2000-1F916`	234 chars!

Not complete sets (below 50%)

Total characters = 17. This means that the typeface supports only specific languages/characters from these Unicode blocks.

Set	Range	Notes
Phonetic Extensions	`U+1D00-1D7F`	14 chars
Phonetic Extensions Supplement:	`U+1D80-1DBF`	3 chars

Weights & styles

It has 6 weights with matching italics. It also comes in a variable font format with a weight variation axis that supports weights from 200 to 900.

Debugging notes

Wed, 13 Nov 2019 15:00:00 GMT

You can find here some notes I wrote while watching an excellent talk by Rebecca Hill: Essential JavaScript debugging tools for the modern detective

`$` (dollar) selector

The tips in this section are for the console tab of Chrome DevTools.

$ is an alias for document.querySelector:

// the following statements are equivalent:
$("#an-id");
document.querySelector("#an-id");

$$ is an alias for document.querySelectorAll:

// the following statements are equivalent:
$$("p");
document.querySelectorAll("p");

$0 returns the most recent HTML element you inspected; $1 the second most recent and so forth until $4. Console Utilities API Reference: \$0-\$4

Logging stuff to the console

Prefer console.log({ variable })—wrap the variable inside an object—instead of console.log(variable) to get a little more context:
```
var number = 1;
console.log(number); // prints: 1
console.log({ number }); // prints { number: 1 }
```

console.table makes more readable the logs for your arrays of objects:

const users = [
  { name: "John", age: 26 },
  { name: "Jane", age: 39 },
];
console.table(users);

The code above prints:

Results from console.table

console.group creates collapsable groups of console logs:
```
console.group("Greetings");
console.log("Hello");
console.log("World");
console.groupEnd();
```
The statements above prints:

Results from console.group/groupEnd
Use console.trace to quick log the call stack.
Use console.dir if you want to inspect an HTML element as an object instead of logging its markup with console.log. Consider the following examples:
console.log() prints:

Results from console.log

console.dir prints:

Results from console.dir

console.dir is even more useful in Node because it adds colors, and you can inspect nested objects instead of only logging something like: [Object]. Consider the following example:

const user = {
  name: "John",
  age: 26,
  pets: [
    { name: "Azor", type: "dog", favouriteOwner: { name: "Sally" } },
    { name: "Kitty", type: "cat" },
  ],
};

console.log(user); // prints:
// {
//   name: 'John',
//   age: 26,
//   pets: [
//     { name: 'Azor', type: 'dog', favouriteOwner: [Object] },
//     { name: 'Kitty', type: 'cat' } ]
// }

const options = { colors: true, depth: null };
console.dir(value, options); // prints:
// {
//   name: 'John',
//   age: 26,
//   pets: [
//     { name: 'Azor', type: 'dog', favouriteOwner: { name: "Sally" } },
//     { name: 'Kitty', type: 'cat' } ]
// }

To make your life easier, find or create VS Code snippets for logging stuff into console. ES7 snippets extension for VS Code has some console snippets:

Prefix	Method
`clg→`	`console.log(object)`
`clo→`	`console.log("object", object)`
`ctm→`	`console.time("timeId")`
`cte→`	`console.timeEnd("timeId")`
`cas→`	`console.assert(expression,object)`
`ccl→`	`console.clear()`
`cco→`	`console.count(label)`
`cdi→`	`console.dir`
`cer→`	`console.error(object)`
`cgr→`	`console.group(label)`
`cge→`	`console.groupEnd()`
`ctr→`	`console.trace(object)`
`cwa→`	`console.warn`
`cin→`	`console.info`

Debugging

You can debug your JavaScript applications with VS Code’s Debugger for Chrome extension.

CRA debug:

For CRA app you need the following .vscode/launch.json config:

.vscode/launch.json

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "type": "chrome",
      "request": "launch",
      "name": "CRA Debug",
      "url": "http://localhost:3000",
      "webRoot": "${workspaceFolder}/src",
      "sourceMapPathOverrides": {
        "webpack:///src/*": "${webRoot}/*"
      }
    }
  ]
}

You don’t have to specify the sourceMapPathOverrides explicitly because the default value includes what we specify in the above code snippet.

Gatsby debug

If you want to debug Gatsby’s development build, you need a similar launch.json config, but you’ll have to change the port from 3000 to 8000.

You also have to override the Webpack configuration (at the time of writing at least) in develop stage to include the eval-source-map devtool. Otherwise, the line numbering will be messed up, and you won’t be able to add breakpoints. Relevant GitHub issue

gatsby-node.js

exports.onCreateWebpackConfig = ({ stage, actions }) => {
  if (stage === "develop") {
    actions.setWebpackConfig({
      devtool: "eval-source-map",
    });
  }
};

You can add breakpoints to DOM nodes by right-clicking and selecting “break on” under the Elements tab inside DevTools. For example, you can add breakpoints to monitor an attribute change or when the DOM node gets removed.
You can add conditional breakpoints (based on a known variable): @ 11:47 in the video.
The “pause on all/uncaught exceptions” can be useful.
Use “Blackbox script” to stop stepping into certain scripts. For example, black box an external library script because the bug you’re investigating is more likely to be inside your codebase, not in the external library. You can enable that by right-clicking in a file under the Sources tab and selecting “Blackbox script”.

DevTools snippets @ 14:20

DevTools Snippets

Useful Lodash utilities

Sun, 13 Oct 2019 15:06:00 GMT

In this post, you can find a collection of the most useful lodash utilities. Each method has a quick description, its signature, and examples on how to use it. It also has links to the documentation, the weekly downloads (from NPM), and the bundle size from bundlephobia.

The goal here is to list as many methods as possible, in the least possible space. I tried not to include methods that are identical to their native counterparts unless they are popular. You can’t say that this post is something new or groundbreaking, but you can rather see it as a resource that will remind/introduce you to cool lodash methods quickly. To give you an idea, most of the examples are from the lodash documentation. The descriptions, on the other hand, are custom because I wanted to provide clear, non-technical explanations (at least I tried). Finally, I assume that you download the individual packages, not the whole library. As a result, you won’t see me calling the methods with the underscore e.g. _.map() but directly e.g. map().

* You’ll see that many functions have a parameter named iteratee with a default value of identity. Both of them are helper lodash functions listed under the Utils section. Click the links above to learn more.

Array: chunk(), flatten(), compact(), set methods, zip().
Collection: countBy(), groupBy(), orderBy(), partition(), popular with ES6 alternatives, other collection methods.
Math: maxBy(), minBy(), round().
String: truncate(), pad(), deburr(), escape() - unescape(), capitalize(), change letter casing, other sting methods.
Util: identity() - iteratee(), bindAll(), flow(), times(), range(), other util methods.
Lang: clone(), cloneDeep(), isEqual(), isEmpty(), type checking, other type-checking methods, type conversion methods.
Seq
Object: get() - set(), has() - hasIn(), assign() - assignIn(), merge(), defaults() - defaultsDeep(), pick() - omit(), toPairs() - fromPairs(), keys() - values(), other object methods.
Function: debounce() - throttle(), once(), curry() - partial(), bind(), other function methods.
Links

Array

Methods to chunk, flatten, and perform set operations on arrays.

chunk

chunk(array, chunkSize = 1): Split an array into equal chunks. Docs - weekly downloads - size.

chunk(["a", "b", "c", "d"], 2); // [["a", "b"], ["c", "d"]]
chunk(["a", "b", "c", "d"], 3); // [["a", "b", "c"], ["d"]]

flatten - flattenDeep - flattenDepth

flatten(array): Flatten a nested array by 1 level. Docs - weekly downloads - size.
flattenDeep(array): Flatten the array recursively. Docs - weekly downloads - size.
flattenDepth(array, depth = 1): Specify the levels. Docs - weekly downloads - size.

const nested = [1, [2, [3, [4]], 5]];
flatten(nested); // [1, 2, [3, [4]], 5]
flattenDeep(nested); // [1, 2, 3, 4, 5]
flattenDepth(nested, 1); // [1, 2, [3, [4]], 5]
flattenDepth(nested, 2); // [1, 2, 3, [4], 5]

compact

compact(array): Remove the falsey values. Docs - weekly downloads - size.

compact([0, 1, false, 2, "", 3]); // [1, 2, 3]

Sets

I think it’s easier to understand what each method does in this section by looking at the pictures, rather than reading the descriptions. Set images are from Wikipedia.

Set methods in lodash

union

union(arrays): Create an array that holds all the unique values from both arrays. Docs - weekly downloads - size.
unionBy(arrays, iteratee = identity): It’s like union, but you can specify the unique criterion. Docs - weekly downloads - size.

union([2, 1], [2, 3]); // [2, 1, 3]
unionBy([{ x: 1 }], [{ x: 2 }, { x: 1 }], "x"); // [{ x: 1 }, { x: 2 }]

intersection

intersection(arrays): Create an array that holds only the unique values that exist in both arrays. Docs - weekly downloads - size.
intersectionBy(arrays, iteratee = identity): It’s like intersection, but you can specify the unique criterion. Docs - weekly downloads - size.

intersection([2, 1], [2, 3]); // [2]
intersectionBy([{ x: 1 }], [{ x: 2 }, { x: 1 }], "x"); // [{ x: 1 }]

difference

difference(arrays): Create an array that holds the unique values that exist in the first array but not in the second. Docs - weekly downloads - size.
differenceBy(arrays, iteratee = identity): It’s like difference, but you can specify the unique criterion. Docs - weekly downloads - size.

difference([2, 1], [2, 3]); // [1]
differenceBy([{ x: 2 }, { x: 1 }], [{ x: 1 }], "x"); // [{ x: 2 }]

See also without if you prefer to pass values instead of a second array.

xor

xor(arrays): Create an array that holds the unique values from both arrays but not those that exist in both. Docs - weekly downloads - size.
xorBy(arrays, iteratee = identity): It’s like xor, but you can specify the unique criterion. Docs - weekly downloads - size.

xor([2, 1], [2, 3]); // [1, 3]
xorBy([{ x: 1 }], [{ x: 2 }, { x: 1 }], "x"); // [{ x: 2 }]

zip

zip(arrays): Group arrays, see the example. Docs - weekly downloads - size.

zip(["a", "b"], [1, 2], [true, false]); // [["a", 1, true], ["b", 2, false]]
// edge cases:
zip(["a", "b"], [1, 2, 3]); // [["a", 1], ["b", 2], [undefined, 3]], different length.
zip(["a", "b"]); // [["a"], ["b"]], single array.

Collection

You can use these methods for array-like objects and regular objects.

Example users array

I will reference this array in some examples.

const users = [
  { name: "barney", age: 36, active: false },
  { name: "fred", age: 40, active: true },
  { name: "pebbles", age: 1, active: false },
];

countBy

countBy(collection, iteratee = identity): Count the different values. Docs - weekly downloads - size.

countBy([1, 1, 2]); // { "1": 2, "2": 1 }
countBy([6.1, 4.2, 6.3], Math.floor); // { "4": 1, "6": 2 }
countBy(["one", "two", "three"], "length"); // { "3": 2, "5": 1 }
// 2 items with length 3, 1 item with length 5.
countBy(users, "active"); // { false: 2, true: 1 }, by an object property.

groupBy

groupBy(collection, iteratee = identity): Group the different values. Example users array - Docs - weekly downloads - size.

groupBy([6.1, 4.2, 6.3], Math.floor); // { "4": [4.2], "6": [6.1, 6.3] }
groupBy(["one", "two", "three"], "length"); // { "3": ["one", "two"], "5": ["three"] }
groupBy(users, "active"); // { "false": [barney, pebbles], "true": [fred] }

orderBy

orderBy(collection, iteratees = identity, [order]): Sort the collection by a property. It’s like sortBy, but you can also set the order (asc/desc). Example users array - Docs - weekly downloads - size.

orderBy(users, "age", "desc"); // [fred (40), barney (36)], pebbles (1)]
orderBy(users, ["name", "age"], ["asc", "desc"]); // by name and then (if equal) by age.
// [barney("b",36), fred("f",40), pebbles("p",1)]

partition

partition(collection, predicate = identity): Split the collection into two groups. Example users array - Docs - weekly downloads - size.

partition(users, (o) => o.active); // [[fred(true)], [barney(false), pebbles(false)]], func.
partition(users, { age: 1, active: false }); // [[pebbles], [barney, fred]], object.
partition(users, ["active", false]); // [[barney, pebbles], [fred]], array.
partition(users, "active"); // [[fred], [barney, pebbles]], string.

Popular methods with ES6 alternatives

These methods are useful if you want to support Internet Explorer—instead of relying on polyfills and native methods—or if you want to use the iteratee shorthand. It has been also reported in the past that they are more performant than the native methods, but I can’t find a definitive resource for that.

includes(collection, value, fromIndex = 0) Docs - weekly downloads - size
each/foreach(collection, iteratee = identity) Docs - weekly downloads - size
map(collection, iteratee = identity) Docs - weekly downloads - size
filter(collection, predicate= identity) Docs - weekly downloads - size
find(collection, predicate = identity, fromIndex = 0) Docs - weekly downloads - size
reduce(collection, iteratee = identity, accumulator) Docs - weekly downloads - size

Example users array

/* 1 */ includes([1, 2, 3], 1); // true
/* 2 */ forEach([1, 2], (value) => console.log(value)); // Logs 1 then 2.
/* 3 */ map([4, 8], (n) => n * 2); // [8, 16]
/* 3 */ map(users, "name"); // ["barney", "fred", "pebbles"]
/* 4 */ filter(users, { age: 36, active: false }); // [barney]
/* 4 */ filter(users, ["active", false]); // [barney, pebbles]
/* 5 */ find(users, ["active", false]); // [barney], the first it finds.
/* 6 */ reduce([1, 2, 3], (acc, next) => (acc += next), 0); // 6

Other collection methods

shuffle(collection) Shuffle the elements. Docs - weekly downloads - size
sample(collection) Get a random element. Docs - weekly downloads - size

shuffle([1, 2, 3, 4]); // [4, 1, 3, 2]
sample([1, 2, 3, 4]); // 2, get a random element.

Math

These are three of the most interesting lodash math methods.

maxBy

maxBy(array, iteratee = identity): Find the max in an array. Example users array - Docs - weekly downloads - size.

maxBy(users, (o) => o.age); // fred object, function
maxBy(users, "age"); // fred object, shorthand

minBy

minBy(array, iteratee = identity): Find the min in an array. Example users array - Docs - weekly downloads - size.

minBy(users, (o) => o.age); // pebbles object, function
minBy(users, "age"); // pebbles object, shorthand

round

round(number, precision = 0): It’s like Math.round but with precision. Docs - weekly downloads - size.

round(4.006); // 4
round(4.006, 2); // 4.01
round(4.004, 2); // 4
round(4060, -2); // 4100, 60 > 50 (middle with 2 precision)
round(4060, -3); // 4000, 60 < 500 (middle with 3 precision)

String

Methods to truncate, pad, escape characters, and change the casing of a string.

truncate

Shorten a string to a certain length: Docs - weekly downloads - size.

truncate(string, { length = 30, omission = "...", [separator] });

const body = "hi-diddly-ho there, neighborino";
truncate(body); // "hi-diddly-ho there, neighbo..."
truncate(body, { length: 24, separator: " " }); // "hi-diddly-ho there,..."
truncate(body, { length: 24, separator: /,? +/ }); // "hi-diddly-ho there..."
truncate(body, { omission: " [...]" }); // "hi-diddly-ho there, neig [...]"

pad

pad(string, length = 0, padCharacters): Adds a character before and after the string. See also padStart and padEnd. Docs - weekly downloads - size.

pad("hello", 10); // "  hello   ", 2 spaces at the start, 3 at the end.
pad("mark", 10, "_"); // "___mark___"
padStart(342, 10, "."); // ".......342", useful in CLI tools.
padStart(12, 10, "."); //  "........12"

deburr

deburr(string): Removes combining diacritical marks, and converts Latin-1 Supplement and Latin Extended-A letters to basic Latin letters. Docs - weekly downloads - size.

deburr("déjà vu"); // "deja vu"

escape - unescape

escape(string): Convert & < > '' ' to & < > " '. In other words, convert characters to HTML entities. Docs - weekly downloads - size.
unescape(string): The opposite of escape, convert HTML entities to characters. Docs - weekly downloads - size.

escape("fred, barney, & pebbles"); // "fred, barney, &amp; pebbles"
unescape("fred, barney, &amp; pebbles"); // "fred, barney, & pebbles"

capitalize

capitalize(string): Capitalize a string. Docs - weekly downloads - size.

capitalize("mark"); // Mark
capitalize("MARK"); // Mark
capitalize("mArK"); // Mark

Change letter casing

cameCase(string): Docs - weekly downloads - size.
kebabCase(string): Docs - weekly downloads - size.
snakeCase(string): Docs - weekly downloads - size.
startCase(string): Docs - weekly downloads - size.

camelCase("--foo-bar--"); // "fooBar"
camelCase("__FOO_BAR__"); // "fooBar"
camelCase("Foo Bar"); 
kebabCase("Foo Bar"); 
snakeCase("Foo Bar"); 
startCase("Foo Bar");

Other popular string methods

template: It’s popular but not really relevant now that we have template literals. Docs - weekly downloads - size.

Util

General functional programming utilities.

identity and iteratee

identity(value): A helper utility that returns the first value it receives. It’s used as a default value for the iteratee in many methods. It’s like the constant method. Docs - weekly downloads - size
iteratee(func = identity): Creates a new function. If the func argument is a function, the new function calls func with the arguments it receives—doesn’t sound really useful. When func is a property name (e.g. user.age), though, the new function returns the value for that property. Finally, if func is an array or object, it returns true for elements that have those properties, or false otherwise. Consider the examples below. Docs - weekly downloads - size

const users = [
  { name: "barney", age: 36, active: true },
  { name: "fred", age: 40, active: false },
];
identity(users[0]) === users[0]; // true
filter(users, iteratee({ name: "barney" })); // [barney], `matches` shorthand.
filter(users, iteratee(["name", "fred"])); // [fred], `matchesProperty` shorthand.
map(users, iteratee("name")); // ["barney", "fred"], `property` shorthand.

See also matches, matchesProperty, property, and method.

bindAll

bindAll(object, methodNames): For the object methods you choose, this inside them points to the object, no matter how they’re called. Docs - weekly downloads - size

const view = {
  label: "docs",
  click: function () {
    console.log("clicked " + this.label);
  },
};
bindAll(view, ["click"]);
element.addEventListener("click", view.click); // "clicked docs"

flow

flow(funcs): Chain functions from left to right. See also flowRight. Docs - weekly downloads - size

const add = (a, b) => a + b;
const square = (n) => n * n;
const addSquare = flow([add, square]); // or flow(add, square) if you prefer.
addSquare(1, 2); // returns 9. Returns 3 after `add` and 9 after `square`

Lodash FP and currying can make flow even more interesting.

times

times(n, iteratee = identity): Call the iteratee n times—passing the current index each time—and return an array with the results. Docs - weekly downloads - size

times(3); // [0, 1, 2]
times(3, String); // ["0", "1", "2"]
times(4, constant(0)); // [0, 0, 0, 0]

range - rangeRight

Create an array where you specify the start, end, and the step. The start number is optional.

range(start = 0, end, step = 1): Docs - weekly downloads - size
rangeRight(start = 0, end, step = 1): Docs - weekly downloads - size

range(4); // [0, 1, 2, 3]
range(-4); // [0, -1, -2, -3]
range(0, 20, 5); // [0, 5, 10, 15]
range(1, 4, 0); // [1, 1, 1]
range(0); // []
rangeRight(4); // [3, 2, 1, 0]
rangeRight(-4); // [-3, -2, -1, 0]

Other interesting methods

cond(pairs) Ιt’s like a switch statement, but you create it on the fly by composing functions. See also toPairs. Docs - weekly downloads - size
conforms(source) Similar to the object shorthand of the iteratee but you can specify the operation. Docs - weekly downloads - size
noop() A method that returns undefined. Useful when you want to call a method no matter what, without getting an error. See the example. Docs - weekly downloads - size
uniqueId(prefix = '') Generate a unique ID with an optional prefix. Docs - weekly downloads - size
over(iteratees = identity) Docs - weekly downloads - size

Example users array

const func = cond([
  [matches({ a: 1 }), /* . */ constant("matches A")],
  [conforms({ b: isNumber }), constant("matches B")],
  [stubTrue, /* .......... */ constant("no match")],
]);
/* 1 */ func({ a: 1, b: 2 }); // "matches A"
/* 1 */ func({ a: 0, b: 1 }); // "matches B"
/* 1 */ func({ a: "1", b: "2" }); // "no match"
/* 2 */ filter(
  users,
  conforms({ age: (n) => n > 1, active: (n) => n === true })
); // [fred]
/* 3 */ times(2, noop); // [undefined, undefined]
/* 3 */ const callbackThatMustBeCalled = userCallback || noop;
/* 4 */ uniqueId(); // "105"
/* 4 */ uniqueId("contact_"); // "contact_104"
/* 5 */ const func = over([Math.max, Math.min]);
/* 5 */ func(1, 2, 3, 4); // [4, 1]

Lang

Utilities that simplify cloning objects, type checking, and type conversions.

clone

clone(value): Create a shallow copy of a value. Docs - weekly downloads - size

const objects = [{ a: 1 }, { b: 2 }];
const shallow = clone(objects);
shallow[0] === objects[0]; // true

cloneDeep

cloneDeep(value): Create a deep copy of a value. Docs - weekly downloads - size

const objects = [{ a: 1 }, { b: 2 }];
const deep = cloneDeep(objects);
deep[0] === objects[0]; // false

isEqual

isEqual(value, other): Deep comparison between two values. Docs - weekly downloads - size

const object = { a: 1 };
const other = { a: 1 };
isEqual(object, other); // true
object === other; // false

isEmpty

isEmpty(value): Check if a value is empty. For array-like: length is 0; for objects: no own properties; for maps/sets: size is 0. Docs - weekly downloads - size

isEmpty(null); // true
isEmpty(true); // true
isEmpty(1); // true
isEmpty([1, 2, 3]); // false
isEmpty({ a: 1 }); // false

Type-checking

isNumber

isNumber(value): Check if the value is a Number primitive or object. Docs - weekly downloads - size

isNumber(3); // true
isNumber(Number.MIN_VALUE); // true
isNumber(Infinity); // true
isNumber("3"); // false

isInteger

isInteger(value): Docs - weekly downloads - size

isInteger(3); // true
isInteger(Number.MIN_VALUE); // false
isInteger(Infinity); // false
isInteger("3"); // false
isInteger(3.2); // false

isPlainObject

isPlainObject(value): Checks if value is a plain object, that is, an object created by the Object constructor or one with a [[Prototype]] of null. Docs - weekly downloads - size

function Foo() {
  this.a = 1;
}
isPlainObject(new Foo()); // false
isPlainObject([1, 2, 3]); // false
isPlainObject({ x: 0, y: 0 }); // true
isPlainObject(Object.create(null)); // true

isMatch

isMatch(object, source): See if the object has a property with that value. If you partially apply the source, it’s the same as matches. Docs - weekly downloads - size

const object = { a: 1, b: 2 };
isMatch(object, { b: 2 }); // true
isMatch(object, { b: 1 }); // false

Other interesting type-checking methods

isArray, isArrayLike, isBoolean, isDate, isElement, isError, isFinite, isFunction, isNative, isObject, isObjectLike, isRegExp, isSafeInteger, isString, isTypedArray, and isUndefined.

Type conversion methods

toArray, toFinite, toInteger, toLength, toNumber, toPlainObject, toSafeInteger, and toString.

Seq

Chain lodash methods with the Sequence methods. An alternative is the flow method.

Example users array

const youngest = _.chain(users) // start the chain by passing a value.
  .sortBy("age")
  .map((o) => `${o.name} is ${o.age}`)
  .head() // get the first element from the array.
  .value(); // "pebbles is 1", get the value.

See also tap, thru. They both intercept the chain and can modify the value, but thru returns the value.

Object

Methods that simplify object creation, validation, getting/setting properties, and more.

get - set

get(object, property, [defaultValue]): Get the object property, without throwing reference errors if it doesn’t exist. Docs - weekly downloads - size
set(object, property, value): Set an object property. Docs - weekly downloads - size

const object = { a: [{ b: { c: 3 } }] };
get(object, "a[0].b.c"); // 3
get(object, ["a", "0", "b", "c"]); // 3
get(object, "a.b.c", "default"); // 'default'
set(object, "a[0].b.c", 4); // { a: [{ b: { c: 4 } }] }
set(object, ["x", "0", "y", "z"], 5); // { a: [{ b: { c: 4 } }], x: [{ y: { z: 5 } }] }

has - hasIn

has(object, property): Check if the object directly has the property, excluding inherited methods. Docs - weekly downloads - size
hasIn(object, property): Check if the object has the property, directly or by inheritance. Docs - weekly downloads - size

const object = { a: { b: 2 } };
const other = Object.create({ a: Object.create({ b: 2 }) }); // inheritance
has(object, "a"); // true
has(object, "a.b"); // true
has(object, ["a", "b"]); // true
has(other, "a"); // false, a is not a direct property of other.
hasIn(other, "a"); // true

assign - assignIn

assign(object, otherObjects): Assign the own properties of the otherObjects to the object. Docs - weekly downloads - size
assignIn(object, otherObjects): Assign the own and inherited properties of the otherObjects to the object. Docs - weekly downloads - size

function Foo() {
  this.a = 1;
}
function Bar() {
  this.c = 3;
}
Foo.prototype.b = 2;
Bar.prototype.d = 4;
assign({ a: 0 }, new Foo(), new Bar()); // { a: 1, c: 3 }
assignIn({ a: 0 }, new Foo(), new Bar()); // { a: 1, b: 2, c: 3, d: 4 }

merge

merge(object, otherObjects): It’s like assignIn but can merge nested object properties too. See also mergeWith. Docs - weekly downloads - size

const object = { a: [{ b: 2 }, { d: 4 }] };
const otherr = { a: [{ c: 3 }, { e: 5 }] };
merge(object, otherr); //    { a: [{ b: 2, c: 3 }, { d: 4, e: 5 }] }
assignIn(object, otherr); // { a: [{ c: 3 }, { e: 5 }] }
assign(object, otherr); //   { a: [{ c: 3 }, { e: 5 }] }

defaults - defaultsDeep

defaults(object, defaultProperties): If the object doesn’t already have the default properties, add them. Docs - weekly downloads - size
defaultsDeep(object, defaultProperties): Same as defaults but recursively for nested properties too. Docs - weekly downloads - size

defaults({ a: 1 }, { b: 2 }, { a: 3 }); // { a: 1, b: 2 }
defaultsDeep({ a: { b: 2 } }, { a: { b: 1, c: 3 } }); // { a: { b: 2, c: 3 } }

pick - omit

pick(object, properties): Create an object and choose the properties you want. Docs - weekly downloads - size
omit(object, properties): Create an object and choose what properties you don’t want. (It’s slower than pick) Docs - weekly downloads - size

const object = { a: 1, b: "2", c: 3 };
pick(object, ["a", "c"]); // { a: 1, c: 3 }
omit(object, ["a", "c"]); // { b: "2" }

toPairs - fromPairs

toPairs(object): Put the object’s own key-value pairs in an array. Each entry (pair) is also an array. Docs - weekly downloads - size.
fromPairs(pairs): The opposite from toPairs (this is an Array method by the way). Docs - weekly downloads - size.

toPairs({ a: 1, b: 2 }); // [["a", 1], ["b", 2]]
fromPairs([
  ["a", 1],
  ["b", 2],
]); // { a: 1, b: 2 }

keys - values

keys(object): Create an array with the object’s own keys. For inherited properties see keysIn. Docs - weekly downloads - size
values(object): Create an array with the object’s own values. For inherited values see valuesIn. Docs - weekly downloads - size

function Foo() {
  this.a = 1;
  this.b = 2;
}
Foo.prototype.c = 3;
keys(new Foo()); // ["a", "b"]
keys("hi"); // ["0", "1"]
keysIn(new Foo()); // ["a", "b", "c"]
values(new Foo()); // [1, 2]
values("hi"); // ["h", "i"]
valuesIn(new Foo()); //  [1, 2, 3]

Other object methods

create(prototype, optionalProperties) Create an object that inherits from (is linked to) the prototype object (first parameter). Docs - weekly downloads - size
invert(object) Invert the object’s key-value pairs. Docs - weekly downloads - size
invoke(object, property, [args]) Invoke an object’s method. You can also pass arguments. Docs - weekly downloads - size
mapValues(object, iteratee = identity) Docs - weekly downloads - size
transform Similar to reduce but you can stop the iteration by returning false. Docs - weekly downloads - size

Example users array

/* 2 */ const object = { a: 1, b: 2, c: 1 };
/* 2 */ invert(object); // { 1: "c", 2: "b" }
/* 3 */ const object = { a: [{ b: { c: [1, 2, 3, 4] } }] };
/* 3 */ invoke(object, "a[0].b.c.slice", 1, 3); // [2, 3]
/* 4 */ mapValues(users, "age"); // { '0': 36, '1': 40, '2': 1 }
/* 5 */ transform(
  [2, 3, 4],
  function (result, n) {
    result.push((n *= n));
    return n % 2 == 0;
  },
  []
); //  [4, 9]

Function

Methods that moderate the execution of functions and some that help with functional programming like currying/partial application.

debounce - throttle

debounce(func, wait = 0, options={ leading = false, trailing = true }): Groups the func calls if they happen in less than wait ms (imagine the calls as a sudden burst of events). By “group” I mean it doesn’t allow the func to execute. Docs - weekly downloads - size
throttle(func, wait = 0, options={ leading = true, trailing = true }): Calls the func at most every wait ms. Unlike debounce, throttle guarantees the execution of the func at least every wait ms. Docs - weekly downloads - size

True for both of them: If leading and trailing options are true, func is called on the trailing edge of the timeout only if the function is called more than once during the wait timeout.

function scrollHandler(e) {
  console.log("Scrolling and rolling");
}
const debounced = _.debounce(scrollHandler, 600);
const throttled = _.throttle(scrollHandler, 600);
document.addEventListener("scroll", scrollHandler); // regular
// document.addEventListener("scroll", debounced); // debounce
// document.addEventListener("scroll", throttled); // throttle

once

once(func): Create a function that can be called only once. Docs - weekly downloads - size

const initialize = once(createApplication);
initialize(); // `createApplication` is invoked
initialize(); // // `createApplication` is not invoked

curry - partial

A blog post that explains the differences between them: Currying versus partial application (with JavaScript code)

curry(func, arity = func.length): Docs - weekly downloads - size
partial(func, parameters): Docs - weekly downloads - size

const regularFunction = (a, b, c) => [a, b, c];
const curried = curry(regularFunction); 
curried(1)(2)(3); 
curried(1, 2)(3); // [1, 2, 3]
curried(1, 2, 3); // [1, 2, 3]
const partiallyApplied = partial(regularFunction, 1, 2); 
partiallyApplied(3);

bind

bind(func, this, optionalArgs): Create a function with your preferred this binding, and optionally pass (or apply) arguments to that function. The difference with Function.prototype.bind is that it doesn’t set the length property. Docs - weekly downloads - size

function greet(greeting, punctuation) {
  return greeting + " " + this.name + punctuation;
}
const user = { name: "fred" };
const bound = bind(greet, user, "hi");
bound("!"); // "hi fred!"

Other function methods

memoize(func, [resolver]) Cache the result of a function. Docs - weekly downloads - size
unary(func) Creates a function that accepts only one argument. Docs - weekly downloads - size
wrap(value, wrapper = identity) Wrap a function within another function (kind of). Docs - weekly downloads - size
after(n, func) Creates a function that invokes func only after the function has been called n times. Docs - weekly downloads - size
before(n, func) Creates a function that can be called at most n - 1 times. Docs - weekly downloads - size

/* 2 */ map(["6", "8", "10"], unary(parseInt)); // [6, 8, 10], what you'd expect
/* 2 */ ["6", "8", "10"].map(parseInt); // [6, NaN, 2], wrong because parseInt(string, radix)
/* 3 */ const p = _.wrap(
  _.escape,
  (func, text) => `<p> + ${func(text)} + </p>`
);
/* 3 */ p("fred, barney, & pebbles"); // "<p>fred, barney, &amp; pebbles</p>"
/* 4 and 5*/
const saves = ["profile", "settings", "messages"];
const logTheFirst2 = _.before(saves.length, function () {
  console.log("Logging the save.");
});
const completionMessage = _.after(saves.length, function () {
  console.log("Done saving!");
});
_.forEach(saves, function () {
  console.log("Doing other things.");
  logTheFirst2();
  completionMessage();
});
// Doing other things. (1)
// Logging the save.
// Doing other things. (2)
// Logging the save.
// Doing other things. (3)
// Done saving!

* If you like the format, I have similar posts for the Array.prototype and the String.prototype.

JavaScript string methods

Tue, 08 Oct 2019 19:00:00 GMT

In this post, you will find useful string methods and examples on how to use them. All strings, whether they are in object form or literals, inherit from the String.prototype object which is where all the string methods come from. This post tries to list as many useful string methods as possible, in the least possible space. It’s not meant to be an extensive reference.

Keep in mind that none of the string methods mutates the original string; they return a new string. Strings in JavaScript are immutable.

slice

str.slice(startIndex, endIndex = str.length)

Returns a portion of the string from start to end index (end not included). String.prototype.slice()

"Hello world".slice(6); // "world"
"Hello world".slice(0, 5); // "Hello"
// for negative indices it counts from the end of the string.
"Hello world".slice(-5); // "world"
"Hello world".slice(0, -6); // "Hello"

split

str.split(splitPattern, limit = all)

Split the string into an array. The characters that match the split pattern are thrown away. String.prototype.split()

"Hello world".split(" "); // ["Hello", "world"], words.
"Hello world".split(""); // ["H", "e", "l", "l", "o", " ", ..., "d"], characters.
"Hello world".split(); // ["Hello world"], as is.
"Hello world".split("o"); // ["Hell", " w", "rld"]
"Hello world".split(/O/i); // ["Hell", " w", "rld"], you can use regex.
"Hello world".split("", 5); // ["H", "e", "l", "l", "o"], 5 is the new array length.

indexOf

str.indexOf(searchString, fromIndex = 0)

Returns the first index if it finds the string, otherwise, it returns -1. String.prototype.indexOf()

"imagine".indexOf("i"); // 0
"imagine".indexOf("w"); // -1, "w" doesn't exist in the string.
"imagine".indexOf("i", 1); // 4, finds the second "i".

charAt

str.charAt(index = 0)

Returns the character at the index. String.prototype.charAt()

"learn".charAt(3); // "r"
"learn".charAt(); // "l", the first character.
"learn".charAt(99); // "", returns an empty string if the index is out of range.

includes

str.includes(string, fromIndex = 0)

See if the string contains the given string. String.prototype.includes()

"learning js".includes("js"); // true
"learning js".includes("JS"); // false, it's case-sensitive.
"learning js".includes("ea", 2); // false

match

str.match(regexp)

See if the string matches the regular expression. If you use the global flag, it returns an array with the matching strings. If you don’t use the global flag, the returned array has some additional properties. If you use both capture groups and the global flag, you won’t get back the capture groups. Array.prototype.match()

"Season 8 was bad".match(/bad/); // without the global flag, notice the extra properties.
// ["bad", index: 13, input: "Season 8 was bad", groups: undefined]
"Season 8 was bad".match(/as/g); // with global flag.
// ["as", "as"]
"Season 8 was bad".match(/\w+(as)/); // capture group, returns 1st match and the groups.
// ["Seas", "as", index: 0, input: "Season 8 was bad", groups: undefined]
"Season 8 was bad".match(/\w+(as)/g); // global and capture groups, the global wins.
// ["Seas", "was"]
"Season 8 was bad".match(/good/);
// null
"Season 8 was bad".match();
// ["", index: 0, input: "Season 8 was bad", groups: undefined]
"Season 8 was bad".match(null);
// null

See String.prototype.matchAll() if you want to use both capture groups and the global flag. Use String.prototype.search() if you want less details, for example, to see only if the pattern exists, and to get the index of the first match. There is also the RegExp.prototype.test() for simple checks and the RegExp.prototype.exec() which is similar to matchAll.

replace

str.replace(pattern, replacementStr)

Find the regex in the string and replace it with the replacement string. The pattern can be a regex or a string. String.prototype.replace()

"Hello there".replace(/there/, "world"); // "Hello world"
"Hello there".replace("Hello", "Hi"); // "Hi there"
"Hello there".replace("e", "3"); // "H3llo there", string pattern = replace the first.
"Hello there".replace(/e/, "3"); // "H3llo there"
"Hello there".replace(/e/g, "3"); // "H3llo th3r3"
"John Doe".replace(/(\w+)\s(\w+)/, "$2, $1"); // Doe, John, see the notice below.

trim

str.trim()

Removes whitespace characters from the start and from the end of the string. Whitespace characters are space, tab, line endings, etc. String.prototype.trim()

"    Hello world.   \n".trim(); // "Hello world."

Use trimStart/trimLeft to remove whitespace characters only from the beginning and trimEnd/trimRight to remove only from the end.

concat

str.concat(stringN)

Merge two or more strings into a new string. The + operator is more performant. String.prototype.concat()

"learn".concat("ing"); // "learning"
"learn".concat("ing", " ", "js"); // "learning js"
"learn".concat(1, "ng"); // "learn1ng", using numbers.

toLowerCase

str.toLowerCase()

Returns the string in lowercase. String.prototype.toLowerCase()

"Hello world".toLowerCase(); // "hello world"

toUpperCase

str.toUpperCase()

Returns the string in uppercase. String.prototype.toUpperCase()

"Hello world".toUpperCase(); // "HELLO WORLD"

startsWith

str.startsWith(searchString, fromIndex = 0)

See if the string starts with the given string. String.prototype.startsWith()

"learning js".startsWith("learn"); // true
"learning js".startsWith("js"); // false
"learning js!".startsWith("learn", 1); // false

endsWith

str.endsWith(searchString, length = str.length)

See if the string ends with the given string. String.prototype.endsWith()

"learning js".endsWith("js"); // true
"learning js".endsWith("JS"); // false, it's case-sensitive.
"learning js!".endsWith("js", 11); // true, we ignore the "!" character here.

The rest string prototype methods

Yargs examples

Tue, 01 Oct 2019 17:16:00 GMT

Yargs is a Node package that parses arguments and generates help menus for your CLI tools. In this post, you’ll see how to configure Yargs to accomplish those tasks with examples. But before we see the examples, it will be useful to explain some CLI terms first.

Commands and options

CLI tools accept two types of arguments: commands and options.

The commands are arguments that don’t have a hyphen (-) in front of them. For example, git clone or git init; clone and init are two different git commands. Use commands if your program is big and performs a wide range of tasks. If your program performs only a specific task, you don’t need commands.
The options, on the other hand, are prefixed with hyphens. With two hyphens you indicate the full name of the option: e.g.--verbose and with one the alias: e.g.-v. Use options to get input from the user and customize the behavior of the program.

Node shebang

Add this at the top of your file:

#!/usr/bin/env node

I will not attempt to explain what a shebang is—I didn’t find a good, non-technical explanation—but, instead, I will tell you what it does. It locates the node executable and allows you to run the script as a binary in Unix systems. For example, you can run the program with ./scipt.js instead of node script.js. There’s also an ESLint rule for the correct usage of shebang.

bin field in package.json

Use package.json’s bin field to provide aliases for your scripts. You can then use those aliases to call your scripts after you install your package globally. You can install your package globally by running npm link or by publishing it. Consider the following package.json:

package.json

{
  "name": "calculate-stuff",
  "bin": {
    "area": "./area.js",
    "circle-area": "./circle-area.js"
  }
}
// Install it globally with `npm link`.
// Run it with `area --width 10 --height 5` or with `circle-area --radius 5`

By specifying the bin fields, you can now call the area script with:

area --width num --height num

and the circle-area script with:

circle-area --radius num

If you don’t provide aliases, you have to call the index script with calculate-stuff and figure out what to do from there.

Special characters

When working with command-line tools, you may encounter some special characters:

The argument is required: <argument>.
This argument is optional: [argument].
Use the pipe operator for name/alias: <name|alias>.
You indicate an array with an ellipsis [arguments…].

Resources (kind of):

Configure an option

Let’s assume that you want to write a program that gets from the user the width and the height of a rectangle, calculates the area, and prints the result in the console:

area.js

// Get the arguments from somewhere.
const { width, height } = getArguments();
console.log(`The area is ${width * height}`);

You can use Yargs to get the width and height arguments from the user and create a help menu. Calculating the area of a rectangle is a trivial task, but the point is to see how to configure Yargs to parse options and create help menus.

You can start by requiring Yargs. By doing that, you now have access to the arguments.

area.js

#!/usr/bin/env node
var { argv } = require("yargs");
console.log(argv);

If you now call the program with:

node area.js --message=hello

or, thanks to the shebang, with:

./area.js --message=hello

The program will log the following object in the console:

{ _: [], message: 'hello', '$0': 'area.js' }

This is already useful but let’s configure it further. You can add the name of the program the user sees when they open the help menu with the scriptName method. A user can see the help menu by calling the program with the --help option.

area.js

#!/usr/bin/env node
var { argv } = require("yargs")
  .scriptName("area");

You can teach the user how to use your tool when they request the help menu. To do that, you can display a helpful message with the usage method. The $0 will be replaced by the script (program) name.

area.js

#!/usr/bin/env node
var { argv } = require("yargs")
  .scriptName("area")
  .usage("Usage: $0 -w num -h num");

You can also show an example of how to use your tool in the help menu with the example method. With the first parameter you specify what user has to type in the console, and with the second what the program should print:

area.js

var { argv } = require("yargs")
  .scriptName("area")
  .usage("Usage: $0 -w num -h num")
  .example(
    "$0 -w 5 -h 6",
    "Returns the area (30) by multiplying the width with the height."
  );

Let’s now start configuring some options. You’ll configure the width (-w) option with the option method. This method takes the option name as the first parameter and a configuration object as the second. You can find all the valid configuration options in the Yargs API page.

The first thing you can do is to add the --width as an alias for the -w option. To do that, you define the alias property of the configuration object:

area.js

#!/usr/bin/env node
var { argv } = require("yargs")
  .scriptName("area")
  .usage("Usage: $0 -w num -h num")
  .example(
    "$0 -w 5 -h 6",
    "Returns the area (30) by multiplying the width with the height."
  )
  .option("w", {
    
    alias: "width",
  });

You can show a help menu message for the width option with the describe property. This message will show up when the user asks for the help menu:

area.js

#!/usr/bin/env node
var { argv } = require("yargs")
  .scriptName("area")
  .usage("Usage: $0 -w num -h num")
  .example(
    "$0 -w 5 -h 6",
    "Returns the area (30) by multiplying the width with the height."
  )
  .option("w", {
    alias: "width",
    describe: "The width of the area.",
  });

You can make the width option required and show an error message when the user forgets to provide it:

area.js

#!/usr/bin/env node
var { argv } = require("yargs")
  .scriptName("area")
  .usage("Usage: $0 -w num -h num")
  .example(
    "$0 -w 5 -h 6",
    "Returns the area (30) by multiplying the width with the height."
  )
  .option("w", {
    alias: "width",
    describe: "The width of the area.",
    demandOption: "The width is required.",
  });

An alternative is to use a default value with the default property instead of requiring the option. For example, you can add the number 5 as the default value for the width with default: 5.

You can provide a type for the option with the type property:

area.js

#!/usr/bin/env node
var { argv } = require("yargs")
  .scriptName("area")
  .usage("Usage: $0 -w num -h num")
  .example(
    "$0 -w 5 -h 6",
    "Returns the area (30) by multiplying the width with the height."
  )
  .option("w", {
    alias: "width",
    describe: "The width of the area.",
    demandOption: "The width is required.",
    type: "number",
  });

Instead of using the type property, you can also use the number property. For example, you can say that number: true. Other type options you can choose from are the boolean, array, and string.

Finally, you can say how many numbers you expect after the width option with the nargs property. In this example, you only want one, so you can express that with: nargs: 1. If the user calls the program with -w 10 11, only the first value (10) will be assigned to the width:

area.js

#!/usr/bin/env node
var { argv } = require("yargs")
  .scriptName("area")
  .usage("Usage: $0 -w num -h num")
  .example(
    "$0 -w 5 -h 6",
    "Returns the area (30) by multiplying the width with the height."
  )
  .option("w", {
    alias: "width",
    describe: "The width of the area.",
    demandOption: "The width is required.",
    type: "number",
    nargs: 1,
  });

And that’s it for the width option. You can now do something similar for the height option:

area.js

#!/usr/bin/env node
var { argv } = require("yargs")
  .scriptName("area")
  .usage("Usage: $0 -w num -h num")
  .example(
    "$0 -w 5 -h 6",
    "Returns the area (30) by multiplying the width with the height."
  )
  .option("w", {
    alias: "width",
    describe: "The width of the area.",
    demandOption: "The width is required.",
    type: "number",
    nargs: 1,
  })
  .option("h", {
    alias: "height",
    describe: "The height of the area.",
    demandOption: "The height is required.",
    type: "number",
    nargs: 1,
  });

Instead of using the option method, you can use the individual methods to configure the options. One of those methods is the describe method. In the following snippet, I’m using the describe method to override the message of the --help and --version options. More about the individual methods in a bit:

area.js

#!/usr/bin/env node
var { argv } = require("yargs")
  .scriptName("area")
  .usage("Usage: $0 -w num -h num")
  .example(
    "$0 -w 5 -h 6",
    "Returns the area (30) by multiplying the width with the height."
  )
  .option("w", {
    alias: "width",
    describe: "The width of the area.",
    demandOption: "The width is required.",
    type: "number",
    nargs: 1,
  })
  .option("h", {
    alias: "height",
    describe: "The height of the area.",
    demandOption: "The height is required.",
    type: "number",
    nargs: 1,
  })
  .describe("help", "Show help.") 
  .describe("version", "Show version number.");

Finally, you can use the epilog method to print a message at the end of the help menu. In the following example, I show some copyright information.

If everything goes well, you can get the width and height options and print the result in the console:

area.js

#!/usr/bin/env node
var { argv } = require("yargs")
  .scriptName("area")
  .usage("Usage: $0 -w num -h num")
  .example(
    "$0 -w 5 -h 6",
    "Returns the area (30) by multiplying the width with the height."
  )
  .option("w", {
    alias: "width",
    describe: "The width of the area.",
    demandOption: "The width is required.",
    type: "number",
    nargs: 1,
  })
  .option("h", {
    alias: "height",
    describe: "The height of the area.",
    demandOption: "The height is required.",
    type: "number",
    nargs: 1,
  })
  .describe("help", "Show help.") // Override --help usage message.
  .describe("version", "Show version number.") // Override --version usage message.
  .epilog("copyright 2019");

const { width, height } = argv;

console.log(`The area is ${width * height}`);

In the following snippet, you can see what the program prints in the console if you call it with the --help option:

./area.js --help

# Usage: area -w num -h num
#
# Options:
#   --help        Show help.                     [boolean]
#   --version     Show version number.           [boolean]
#   -w, --width   The width of the area.         [number] [required]
#   -h, --height  The height of the area.        [number] [required]
#
# Examples:
#   area -w 5 -h 6  Returns the area (30) by multiplying
#                   the width with the height.
#
# copyright 2019

This is the output if you call it without the --width and --height arguments:

node area.js

# Usage: area -w num -h num
#
# Options:
#   --help        Show help.                     [boolean]
#   --version     Show version number.           [boolean]
#   -w, --width   The width of the area.         [number] [required]
#   -h, --height  The height of the area.        [number] [required]
#
# Examples:
#   area -w 5 -h 6  Returns the area (30) by multiplying
#                   the width with the height.
#
# copyright 2019
#

Use the individual methods

Instead of using the option method, you can use the individual methods. For example, the configuration for the --width option can be written as shown below:

#!/usr/bin/env node
// area.js
var { argv } = require("yargs")
  // ...
  .alias("w", "width")
  .describe("w", "The width of the area.")
  .demandOption("w", "The width is required.")
  // .default("w", 5)
  .number("w")
  .nargs(1)
  .alias("h", "height")
  .describe("h", "The height of the area.")
  .demandOption("h", "The height is required.")
  // .default("h", 5)
  .number("h")
  .nargs(1);
// ...

Run the program

You can now run the program in the following ways:

./area.js -w 5 -h 6, if you add the node shebang at the top of the file.
node area.js -w 5 -h 6.
npm link to install it globally, and then area -w 5 -h 6—replace the area with the name you set up in the bin field of your package.json.

Other interesting configuration options

.implies(). implies('w', 'h') means that if -w has a value, then -h should have a value too. It could be useful in our case because the program needs both options to calculate the area.
.strict(). Throws errors if the options/command requirements are not met. For example, if the type of an option is not correct, it throws an error.
Dealing with quotes

Commands

I don’t need it right now, maybe I will revisit it in the future. For now, see Advanced in Yargs docs.

Links

See the Yargs GitHub page.

Curl examples

Thu, 26 Sep 2019 15:16:00 GMT

curl is a CLI tool that allows you to transfer data to or from a server in a variety of different protocols. In this post, you can find some examples that demonstrate the basic curl options for the HTTP/HTTPS protocols.

Make a GET request

GET is the default HTTP method, so you don’t have to use an option. You can pass only the URL:

curl https://www.google.com

Specify the HTTP Method

You can specify the HTTP method with the -X or --request option. In the following example, I make an OPTIONS request:

curl -X OPTIONS -i https://jsonplaceholder.typicode.com/

The -i or --include option gives back the response headers, not only the response body—more about that in a bit.

1. Perform a POST request

When you perform a POST request, you usually send some data to the server. You can send data with the -d or --data option:

curl https://jsonplaceholder.typicode.com/todos \
     --data "title=Groceries&body=Buy groceries."
# The backslash (\) spans the command over multiple lines

You don’t have to state the method with -X POST because POST is the default method when you use the --data option. You send the data as content-type: application/x-www-form-urlencoded. See also -F or --form and --data-raw.

2. Update with the PUT method

If you want to perform a PUT request, you have to specify the method along with the data you want to send:

curl https://jsonplaceholder.typicode.com/todos/1 \
     -X PUT \
     --data "title=Groceries&body=Buy groceries."

Get the headers

When you make an HTTP request with curl, the default output you get is the response body. If you want to get the headers, you have 3 options:

1. Get both response headers and body

This is what you want most of the times. To get both response headers and body, you can use the -i or --include option:

curl https://www.google.com/ -i

2. Get only the response headers

To get only the response headers—without the response body—use the -I or --head option:

curl https://www.google.com/ -I

With this option, you perform a HEAD HTTP request. You can achieve the same thing if you set the HTTP method to HEAD with curl -X HEAD. But because you get only the response body by default, you also have to include the headers with -i or --include, so the final command would be: curl -X HEAD -i. As a result, the -I or --head option is preferred.

3. Get the request headers too

Sometimes, you want more details about the exchange between the server and the client. For example, you may want to see the request headers. In this case, you can use the -v or --verbose option. You get a lot more information, though, not only the request headers:

curl https://www.google.com/ --verbose

To find the request headers, look for the lines that start with >. The response headers, on the other hand, start with <. The lines that start with an asterisk (*) are TCP or TLS stuff.

Follow redirects

To follow the redirects use the -L or --location option. You should also append the response headers with the -i option to see what’s going on:

curl http://google.com --location -i

Try the same request without the --location option if you want to see the difference.

Save console output to file

You can use the -o or --output option to save the console output in a file:

curl https://www.google.com -i -o google.txt

Save a file

If you want to save a file and keep the original filename in your local machine, use the -O or --remote-name option:

curl -O https://markoskon.com/static/fa71d793057a033cf6b13329754cb6fc/dbbca/all-cool-dev.jpg

If you want to give the file a new name, use the -o or --output instead:

curl -o gatsby-modals.jpg https://markoskon.com/static/fa71d793057a033cf6b13329754cb6fc/dbbca/all-cool-dev.jpg

Advanced

Array prototype methods

Sun, 15 Sep 2019 19:00:00 GMT

In this post, you will find useful JavaScript array methods and some examples on how to use them. In the title I mention array prototype methods instead of array methods; let me explain a bit where this comes from.

You can create an array with the bracket syntax: const arr = [1, 2, 3] or with the Array constructor: const arr = new Array(1, 2, 3). No matter what method you choose, you will get back an object that inherits from the Array. As a result, this object (or this array) will have some handy methods you can use to change the contents of the array or get back an answer. Those are the Array.prototype methods.

The goal of this post is to list as many methods as possible, in the least possible space. It’s not meant to be an extensive reference. If you want to learn more about a method, check the links to the MDN documentation.

Methods that modify the array

Use these on the arguments of a function if you want to trigger people that like functional programming.

1. fill()

arr.fill(value, start = 0, end = arr.length)

Fills the array with the given value from start to end index (end not included). Array.prototype.fill().

// .fill(value, start, end)
[1, 2, 3].fill(0); // [0, 0, 0]
new Array(5).fill(1); // [1, 1, 1, 1, 1]
[1, 2, 3].fill(5, 1); // [1, 5, 5]
[1, 2, 3, 4].fill(5, 1, 3); // [1, 5, 5, 4]

2. reverse()

arr.reverse()

Reverses the array. Array.prototype.reverse().

[1, 2, 3].reverse(); // [3, 2, 1]

3. pop()

arr.pop()

Removes the last element. Array.prototype.pop()

[1, 2, 3].pop(); // [1, 2], returns the removed element: 3

4. shift()

arr.shift()

Removes the first element. Array.prototype.shift()

[1, 2, 3].shift(); // [2, 3], returns 1

5. push()

arr.push(elementN)

Adds the given elements at end of the array. Array.prototype.push()

// .push(element1, ...)
[1, 2, 3].push(4, 5); // [1, 2, 3, 4, 5], returns the new length: 5
[1, 2, 3].push(4); // [1, 2, 3, 4], returns 4

6. unshift()

arr.unshift(elementN)

Adds the given elements at start of the array. Array.prototype.unshift()

// .unshift(element1, ...)
[1, 2, 3].unshift(4, 5); // [4, 5, 1, 2, 3], returns 5 (new length)
[1, 2, 3].unshift(4); // [4, 1, 2, 3], returns 4

7. sort()

arr.sort(compareFunction)

Sorts the array. TODO. Array.prototype.sort()

8. splice()

arr.splice(start, deleteCount = all from start, elementNToAdd)

Can delete and add elements in the array at the same time. Returns the deleted items. Array.prototype.splice()

// .splice(start, deleteCount, element1, ...)
[1, 2, 3].splice(1); // only delete: [1], returns [2, 3]
[1, 2, 3].splice(0, 2); // only delete: [3], returns [1, 2]
[1, 2, 3].splice(2, 0, 4, 5); // only add: [1, 2, 4, 5, 3], returns []
[1, 2, 3].splice(3, 0, 4, 5); // only add: [1, 2, 3, 4, 5], returns []
[1, 2, 3].splice(1, 2, "two", "three"); // delete and add: [1, "two", "three"], returns [2, 3]
[1, 2, 3].splice(1, 99, "two", "three"); // delete and add: [1, "two", "three"], returns [2, 3]

All the mutator methods.

Methods that do not modify the array

These methods return a new array/result without modifying the initial array.

1. concat()

arr.concat(valueN)

Merge two or more arrays into a new array. Array.prototype.concat()

[1, 2].concat([3, 4]); // [1, 2, 3, 4]
[1, 2].concat(3, 4); // [1, 2, 3, 4]
[1, 2].concat(3); // [1, 2, 3]
[1, 2].concat([3, 4], [5, 6]); // [1, 2, 3, 4, 5, 6]

2. join()

.join(separator)

Join array items into a string. Array.prototype.join()

[1, 2, 3].join(); // "1,2,3"
[1, 2, 3].join(""); // "123"
[1, 2, 3].join(" - "); // "1 - 2 - 3"

3. slice()

slice(start = 0, end = array.length)

Return a portion of the array from start to end index (end not included). Array.prototype.slice()

[1, 2, 3, 4, 5].slice(2); // [3, 4, 5]
[1, 2, 3, 4, 5].slice(2, 4); // [3, 4]
[1, 2, 3, 4, 5].slice(2, 5); // [3, 4, 5]

4. includes()

arr.includes(element, fromIndex = 0)

See if the array contains the given value. Array.Prototype.includes.

// .includes(value, index)
[1, 2, 3].includes(2); // true
[1, 2, 3].includes(4); // false
[1, 2, 3, 4, 5].includes(1, 1); // false

5. indexOf()

arr.indexOf(element, fromIndex = 0)

Returns the first index of the given element in the array or -1 if it doesn’t exist. Array.prototype.indexOf(). To get the last index of the element, use Array.prototype.lastIndexOf().

[1, 2, 3].indexOf(2); // 1
[1, 2, 3].indexOf(4); // -1
[2, 2, 2].indexOf(2); // 0
[1, 2, 3, 4, 5].indexOf(2, 3); // -1

All the accessor methods

Methods that iterate the array

Methods that iterate the array and call the given function for every element. None of these methods modifies the initial array. The function you pass as a parameter has the following signature: function(element, index, array), although most of the times, you’ll use only the first parameter: function(element). The only exception is the function for the reduce method which has the following signature: function(accumulator, element, index, array).

1. map()

arr.map(callback)

Returns a new array of equal length that contains the transformed elements. Array.Prototype.map()

[1, 2, 3].map((element) => element * 2); // [2, 4, 6]
[1, 2, 3].map((element) => element.toString()); // ["1", "2", "3"]

2. forEach()

arr.forEach(callback)

Iterates an array and calls the given function; it returns undefined. Array.Prototype.forEach()

// 0: 1
// 1: 2
// 2: 3
[1, 2, 3].forEach((element, index) => {
  console.log(`${index}: ${element}`);
});

3. reduce()

arr.reduce(callback, initialValue)

Iterates the array and returns anything you like; a value, a smaller array, or an equal-length array. Use it when no other method in this section can give what you want. Array.Prototype.reduce()

// .reduce(callback, initialValue)
[1, 2, 3].reduce((acc, element) => element + acc, 0); // 6, the sum of the elements
[1, 2, 3].reduce((acc, element) => {
  if (element % 2 === 0) return acc;
  return acc.concat(element);
}, []); // [1, 3], an array with only the even numbers. Although filter is a better fit here.

4. filter()

arr.filter(callback)

Returns a new array that contains the filtered elements. Array.Prototype.filter()

[1, 2, 3].filter((element) => element > 2); // [3]
[1, 2, 3].filter((element) => element % 2 === 0); // [2], 2 is the only even number.
[1, 2, 3].filter((element) => element < 0); // []

5. find()

arr.find(callback)

Returns the first element it finds, or undefined if it doesn’t find anything. Array.Prototype.find()

[1, 2, 3].find((element) => element > 2); // 3
[1, 2, 3].find((element) => element % 2 === 0); // 2, The number 2 is even
[1, 2, 3].find((element) => element % 2 !== 0); // 1, 1 is the first odd number.
[1, 2, 3].find((element) => element < 0); // undefined

6. every()

arr.every(callback)

Returns true if every element in the array satisfies the check, otherwise, it returns false. Array.Prototype.every()

[1, 2, 3].every((element) => element % 2 !== 0); // false, The array has even numbers too.
[1, 3, 5].every((element) => element % 2 !== 0); // true, The array has only odd numbers.

7. some()

arr.some(callback)

Returns true if it finds at least one element or false otherwise. Array.Prototype.some()

[1, 2, 3].some((element) => element % 2 === 0); // true, Has at least one even number.
[1, 3, 5].some((element) => element % 2 === 0); // false, The array has no even numbers.

All the iteration methods

Every iterator method also takes an optional second parameter the thisArgument, that's used as the this value inside the callback.

TODO: Array.prototype.flat(), Array.prototype.flatMap()

Tail calls, tail-recursive functions, and tail call optimization

Fri, 06 Sep 2019 16:00:00 GMT

Learn what is a tail call, a tail-recursive function, and how the compiler can optimize them. I will start with something that wasn’t obvious for me: a function cannot be a tail call; it can be a tail function. An example of a tail function is a tail-recursive function; a function call can be a tail call.

Although I use the term tail function, you should avoid using it for functions that are not recursive because no one else does.

Tail call

The first question is when a function call is a tail call. The technical explanation says that:

A function call is a tail call when it’s the last action of the caller function.

A different way to put this is the following:

A function call is a tail call when the called function returns the same thing as the function that called it.

I like the last explanation more because it hints a connection with the call-stack, as you’ll see shortly.

Examples of tail calls

In the following example, the function call returnNumber(2) is a tail call because it’s the last thing the return2 function (caller function) does. Or, if you prefer the other explanation, the function call returnNumber(2) is a tail call because it returns the same thing as the return2 function.

function returnNumber(x) {
  return x;
}
function return2() {
  return returnNumber(2);
}
return2();

You can implement the return2 function in a way that the function call returnNumber(x) is not a tail call:

function returnNumber(x) {
  return x;
}
function return2() {
  const spare1 = 1;
  return spare1 + returnNumber(1);
}
return2();

In the example above, the return2 function gives the same result, but, now, the function call returnNumber(1), at line 6, is not a tail call because it returns 1 and the caller function return2 returns 2. Or, if you prefer the technical explanation, the function call returnNumber(1) is not a tail call because it’s not the last action of the caller function return2. The last action of the return2 function it to resolve the expression spare1 + returnNumber(1) and return it. As it seems, returning a value is not considered an action.

For recognizing tail calls in expressions and statements, and for some pitfalls, see checking whether a function call is in a tail position.

Tail functions

Let’s now assume that you can name as tail function a function that contains a tail call. In the first snippet, the function return2 is a tail function, but, in the second snippet, it’s not. As a result, whether a function is a tail function or not, depends on how you implement it. Let’s now see an example of that by using a tail-recursive function and regular recursive function.

The following example shows a tail-recursive function that returns the factorial of a number:

// acc = accumulator which is a fancy name for a variable
// that you use to store a temporary result.
function factorialTail(number, acc = 1) {
  if (number === 0) return acc;
  return factorial(number - 1, acc * number);
}

And here you see a recursive function (without a tail call) that does the same thing:

function factorial(number) {
  if (number === 0) return 1;
  return number * factorial(number - 1);
}

The tail-recursive function stores the result in an accumulator variable and passes it to the next function call. On the other hand, the plain recursive function performs the calculation in an expression, right before returning it.

Tail call optimization

You saw when a function call is tail call and when a function is a tail function. But why this is important? The answer has to do with the call-stack and a trick the compiler does that’s called tail call optimization (TCO).

Tail call optimization happens when the compiler recognizes that a function call is a tail call and reuses the current stack frame, instead of placing a new one on top that will increase the stack size.

If you use JavaScript, it turns out you shouldn't care much because TCO is supported only by Safari—it's deprecated on Chrome.

Instead of getting technical with call-stacks, stack-frames, and local/global variables, I’ll show you a visualization that shows what happens in the call stack when you call factorialTail(4). In the first snippet, the compiler does not perform TCO and in the second it does. I got the idea from the Wikipedia page but changed it a bit.

No tail call optimization → 5 stack frames:

// factorialTail(number, acc = 1)
call factorialTail (4, 1)
  call factorialTail (3, 4)
    call factorialTail (2, 12)
      call factorialTail (1, 24)
        call factorialTail (0, 24)
        return 24
      return 24
    return 24
  return 24
return 24

Tail call optimization → 1 stack frame:

// factorialTail(number, acc = 1)
call factorialTail (4, 1)
replace with factorialTail(3, 4)
replace with factorialTail(2, 12)
replace with factorialTail(1, 24)
replace with factorialTail(0, 24)
return 24

The first snippet shows that new frames are being allocated—indicated by the indentation—when the compiler does not perform tail call optimization. The second snippet shows that when the compiler performs TCO, it reuses the current frame for the new function call, instead of adding a new frame to the call-stack.

Technical explanation problem

The following example is from Wikipedia’s page about tail call. The author says that the a(data) call is in the tail position. By the way, please, give descriptive names to your variables/functions! This makes sense with the second explanation because it returns the same thing as the foo2 function. It doesn’t make sense with the first explanation, though, because the a(data) call is not the last action of the foo2 function. After the a(data) call, the function stores that result in a variable and then returns it.

function foo2(data) {
  var ret = a(data);
  return ret;
}

Puppeteer examples

Mon, 26 Aug 2019 16:00:00 GMT

This post will help you get started with Puppeteer in Node and learn how to perform some common tasks. Let’s start with a recommended structure for your project.

Project Structure

Write your code inside an async IIFE in the index.js:

index.js

(async () => {
  // Your code goes here.
})();

Or create a new file that you will import:

screenshot.js

module.exports = async () => {
  // Your code goes here.
};

Also, a nodemon script can be useful here to re-run your code after you make changes:

package.json

{
  "scripts": {
    "start": "nodemon src/index.js --ignore data"
  }
}

Launch and create a new page

Launch a browser and open a new page:

const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();

Puppeteer class documentation

Emulate a device

You can emulate a device with page.emulate. This is a list with the available devices.

const devices = require("puppeteer/DeviceDescriptors");
await page.emulate(devices["iPhone X"]);

Set page viewport

Instead of emulating a device, you can set the page viewport with page.setViewport:

await page.setViewport({ width: 1920, height: 1080 });

Visit a URL

After you set the device or the viewport, you can visit a URL with page.goto

await page.goto(url, options);

A screenshot script

You can take screenshots of a page with page.screenshot. Below you can find a screenshot script where I put all the previous sections together:

screenshot.js

const puppeteer = require("puppeteer");

module.exports = async ({
  url = "https://example.com",
  filename = null,
  fullPage = false,
  device = null,
  headless = true,
} = {}) => {
  const browser = await puppeteer.launch({ headless });
  const page = await browser.newPage();

  if (typeof device === "string") {
    let devices = require("puppeteer/DeviceDescriptors");
    await page.emulate(devices[device]);
  } else {
    await page.setViewport({ width: 1920, height: 1080 });
  }

  await page.goto(url);
  await page.screenshot({
    path: filename
      ? `screenshots/${filename}.png`
      : `screenshots/${new Date().getTime()}.png`,
    fullPage,
  });

  await browser.close();
};

You should also create the directories that you’ll store the screenshots, somewhere in your code:

index.js

const fs = require("fs");
const mkdirp = require("mkdirp");

// Create directories.
const directories = ["screenshots"];
directories.forEach((dir) => {
  if (!fs.existsSync(dir)) mkdirp(`./${dir}/`);
});

Close the browser

Don’t forget to close the browser after you finish your work:

await browser.close();

Create pdfs

Besides screenshots, you can also create pdfs with page.pdf:

await page.pdf({
  path: filename
    ? `pdfs/${filename}.pdf`
    : `pdfs/${new Date().getTime()}.pdf`,
  format: "A4",
});

Get access to the window object

If you want access to the window or document objects—to scrape some information from the page, for example—you can use the page.evaluate method:

const imageUrls = await page.evaluate(() => {
  const images = document.querySelectorAll("article img");
  const urls = Array.from(images).map(({ src }) => ({ src }));
  return urls;
});

The function you pass to evaluate is not a closure. As a result, it doesn’t have access to variables defined in the parent scope. Because of that, if you want to use an outside variable (a selector, for example) inside the function, you have to pass that variable as an argument to evaluate:

const imageSelector = "article img";

const imageUrls = await page.evaluate((selector) => {
  const images = document.querySelectorAll(selector);
  const urls = Array.from(images).map(({ src }) => ({ src }));
  return urls;
}, imageSelector);

By the way, you can’t pass an external library (get-urls, for example) as an argument. Instead, assign the result of the evaluate to a variable and use the library outside.

Type and click

If you want to type some text inside an input, use the page.type method:

await page.type("#searchbox input", "Headless Chrome");

And if you want to click something, use the page.click method:

await page.click("my-button");

Wait for selector to appear

After you perform an action, you can wait for a selector to appear before you proceed any further. You can do that with the page.waitForSelector method:

await page.waitForSelector("img");

Page events

Handle events emitted by the Page with the page.on or page.once methods:

page.once('load', () => console.log('Page loaded.')

Use on to handle every event and once to handle only the first. Additionally, you can use the page.removeListener() method to remove the listener:

const requestLogger = (request) => console.log(request.url());
page.on("request", requestLogger);
// later
page.removeListener(requestLogger);

Skip requests

You can skip requests by enabling request interception with page.setRequestInterception(true). You then listen for the request event, and you call request.abort(), request.continue(), or request.respond():

await page.setRequestInterception(true);
page.on("request", (request) => {
  if (request.resourceType() === "image") request.abort();
  else request.continue();
});

Page transitions in Gatsby

Sun, 25 Aug 2019 23:00:00 GMT

You can find many examples on how to implement page transitions in Gatsby, but some are now outdated, some use plugins or libraries that overcomplicate things, and none of them implements transitions with some “complex” logic. Let me explain what I mean with the “complex” logic by describing what kind of page transitions I want to create.

I want to slide-up the current screen when I go from the root route / to the posts route /posts/ and slide it down when I go the opposite way. In other words, imagine that the URLs with more paths are positioned lower than those with fewer paths:

Vertical transitions example

Also, when you go through a collection of some posts, for example, I want the screen to slide left or right depending on the order of the posts.

Horizontal transitions example

In any other case, the screen will fade-in. This is just an idea I had that I wanted to implement; I don’t claim that it results in a good user experience, nor I’m suggesting that you should use it in your projects.

Transition Logic

I used a wrapPageElement layout to manage the transitions because it has the following advantages over a regular layout component:

It doesn’t umount when you navigate on a different page. This is useful because I want it to remember the previous pathname.
It has access to the location prop.

I have a function called getAnimation that takes as arguments the current pathname and the previous and returns the correct animation. It also takes an optional MDX array that has the slugs of the posts and their order. The layout calls that method and passes the result in a wrapper component, that you’ll see in the next section.

Using only reveal animations

In the first solution, that’s located in the master branch, I don’t use both reveal and exit animations. I use only reveal animations because I want something minimal. I create a wrapper component that wraps the section I want to animate, and I pass to it an emotion keyframe object—with the animation prop. I then use that animation object inside the component’s CSS. The component animates/renders when the current animation is different from the previous or when the pathname changes—I pass the pathname string to the key prop. This is not exactly the effect I had in mind, but it’s quite pleasing, especially the slide-left/right. I am a fan of those animations because they are not complicated, they are performant, if they only use CSS, and they are not so intrusive. By the way, I do something similar in this blog to fade-in the content area of the pages.

Full page

After that, I wrapped a bigger part of the layout with the wrapper component to animate the whole page, not only the content portion. The result is closer to what I had in mind, but it’s not as smooth as in the previous solution. You can find this solution in the full-page branch.

Using the transition hook from react-spring

But if I wanted to achieve the result you saw in the GIFs, I would have to use both enter and leave animations. The main problem here is that when you animate a page in, you don’t know what the exit animation will be. You also want a library that will create two pages when in fact there is only one. react-spring kind of does all of these with the useTransition hook. There are some bugs though that I didn’t investigate much.

To animate with react-spring, I did the following:

I changed the animations from emotion keyframes to react-spring transition objects. I didn’t change the logic inside the getAnimation function, only some imports.
I removed the wrapper component, and instead, I wrapped the content I wanted to animate with an animated.div. That animated.div is under a transition map in the JSX.

You can find this solution in the react-spring-transition branch.

GraphQL queries inside MDX files

Thu, 22 Aug 2019 15:25:00 GMT

When you want to query data with GraphQL inside your MDX pages, you have the following options:

Default MDX config

If you are using the default configuration of gatsby-plugin-mdx, just export a GraphQL query as you would do in a regular Gatsby page. By default configuration, I mean that you installed gatsby-plugin-mdx, @mdx-js/mdx, and @mdx-js/react, and finally, you added gatsby-plugin-mdx as a plugin in gatsby-config.js. Then, you can access the query data from props.data.yourdata.

In the following example, I render the astronaut image from the default Gatsby starter, and I display the props:

src/pages/example.mdx

# Example

Example MDX page. Here's the image of an astronaut:

import { graphql } from "gatsby";
import Image from "gatsby-image";

export const query = graphql`
  query {
    astronaut: file(relativePath: { regex: "/gatsby-astronaut/i" }) {
      childImageSharp {
        fluid(maxWidth: 600) {
          ...GatsbyImageSharpFluid_tracedSVG
        }
      }
    }
  }
`;


<Image
  title="astronaut"
  fluid={props.data.astronaut.childImageSharp.fluid}
/>

Log the `props.data`:

<pre>{JSON.stringify(props, null, 2)}</pre>

Custom templates

If you create the MDX pages programmatically, and you render them with a custom template, create a new component that uses a StaticQuery, then import that component in your MDX page and render it.

If you try to create a page query inside the MDX file or try to render a StaticQuery, you’ll get the following error:

Error: It appears like Gatsby is misconfigured. Gatsby related GraphQL calls are supposed to only be evaluated at compile time, and then compiled away…

Recommended folder structure:

__ src
____ components
______ AstronautImage.jsx
____ pages
______ index.js
______ example.mdx

The MDX file:

src/pages/example.mdx

# Example

Here's the image of an astronaut:

import AstronautImage from "../components/AstronautImage";

<AstronautImage />

And finally, the custom component that uses the StaticQuery:

src/components/AstronautImage.jsx

import React from "react";
import { graphql, StaticQuery } from "gatsby";
import Image from "gatsby-image";

export default () => (
  <StaticQuery
    query={graphql`
      query {
        astronaut: file(relativePath: { regex: "/gatsby-astronaut/i" }) {
          childImageSharp {
            fluid(maxWidth: 600) {
              ...GatsbyImageSharpFluid_tracedSVG
            }
          }
        }
      }
    `}
    render={(data) => (
      <Image
        title="astronaut"
        fluid={data.astronaut.childImageSharp.fluid}
      />
    )}
  />
);

Extra

In this section you can find an MDX template, a gatsby-node.js file that creates programmatically MDX pages, and some links.

Custom template file

src/templates/MDXTemplate.jsx

import React from "react";
import { graphql } from "gatsby";
import { MDXRenderer } from "gatsby-plugin-mdx";
import Layout from "../components/layout";

export default ({
  data: {
    mdx: { body },
  },
}) => {
  return (
    <Layout>
      <h1>Hello from the template</h1>
      <MDXRenderer>{body}</MDXRenderer>
    </Layout>
  );
};

// Notice the slug field in the MDX type you created
// in onCreateNode.
// The $slug variable comes from the page context
// you passed while creating the page.
export const query = graphql`
  query MDXBySlug($slug: String!) {
    mdx(fields: { slug: { eq: $slug } }) {
      body
    }
  }
`;

gatsby-node.js

const { createFilePath } = require("gatsby-source-filesystem");
const mdxTemplate = require.resolve("./src/templates/MDXTemplate.jsx");

exports.onCreatePage = ({
  page,
  actions: { createPage, deletePage },
}) => {
  const { frontmatter } = page.context;
  // Assuming only MDX pages have a frontmatter field...
  if (frontmatter) {
    deletePage(page);
    createPage({
      ...page,
      component: mdxTemplate,
      context: {
        ...page.context,
        slug: page.path,
      },
    });
  }
};

exports.onCreateNode = ({ node, actions, getNode }) => {
  const { createNodeField } = actions;
  if (node.internal.type === "Mdx") {
    const value = createFilePath({ node, getNode });
    createNodeField({
      name: "slug",
      node,
      value,
    });
  }
};

wrapRootElement vs wrapPageElement

Wed, 21 Aug 2019 12:00:00 GMT

According to Gatsby documentation, you can use wrapRootElement to wrap your application with provider components and wrapPageElement to wrap your pages with components that won’t get unmounted on page change. But that explanation begs the question: why we don’t do both with wrapPageElement, and instead, we need two separate APIs to accomplish those tasks? Let’s see their differences to understand why.

Differences

First of all, wrapPageElement is a child of wrapRootElement as you can see in the following image:

wrapRootElement vs wrapPageElement

Additionally, wrapPageElement is a child of the router—wrapRootElement is not—so it has access to the router props. You can access those props if you destructure the props field from the first argument:

gatsby-browser.js|gatsby-ssr.js

exports.wrapPageElement = ({ element, props }) => {
  return <Layout {...props}>{element}</Layout>;
};

In the following snippet you can see an example of that props field:

{
  "path": "/",
  "location": {
    "href": "http://localhost:8001/",
    "ancestorOrigins": {},
    "origin": "http://localhost:8001",
    "protocol": "http:",
    "host": "localhost:8001",
    "hostname": "localhost",
    "port": "8001",
    "pathname": "/",
    "search": "",
    "hash": "",
    "state": null,
    "key": "initial"
  },
  "pageResources": {
    "json": {
      "pageContext": {
        "isCreatedByStatefulCreatePages": true
      }
    },
    "page": {
      "componentChunkName": "component---src-pages-index-js",
      "path": "/",
      "webpackCompilationHash": "4987305ccd829dc361b7"
    }
  },
  "uri": "/",
  "pageContext": {
    "isCreatedByStatefulCreatePages": true
  },
  "pathContext": {
    "isCreatedByStatefulCreatePages": true
  }
}

As a result, wrapPageElement renders every time the page changes—wrapRootElement does not—making it ideal for complex page transitions, or for stuff that need the page path, like an internationalization context provider for example. On the other hand, because wrapRootElement doesn’t render when the page changes, it’s a good fit for context providers that don’t need the page, like theme or global application state providers. And that’s their biggest difference.

One similarity they share is that they both mount only once, as opposed to a regular Layout component—that you use inside your pages—that will unmount every time the page changes.

Finally, if you don’t provide an implementation for wrapPageElement in gatsby-ssr.js, it will use the implementation from gatsby-browser.js. This is not true for wrapRootElement.

Conclusion

Coming back to the initial question, yes, you can use wrapPageElement element for everything, but it’s better to use it only for providers that need the router props, or for page transition layouts, and use wrapRootElement for any other provider, like theme and global state providers.

Extra

Here I list some links and the code I used to find the differences.

Code

The gatsby-browser.js file:

gatsby-browser.js

import React from "react";
import Layout from "./src/components/Layout";

export const wrapRootElement = ({ element, ...restProps }, ...args) => {
  return (
    <Layout
      name="wrapRootElement"
      props={restProps}
      args={args}
      mode="browser"
    >
      {element}
    </Layout>
  );
};
export const wrapPageElement = ({ element, ...restProps }, ...args) => {
  return (
    // <Layout name="wrapPageElement" props={{}} args={args} mode="browser">
    <Layout
      name="wrapPageElement"
      props={restProps}
      args={args}
      mode="browser"
    >
      {element}
    </Layout>
  );
};

And the Layout component:

src/components/Layout.jsx

import React, { useEffect } from "react";

const Layout = ({ name, props, args, mode, children }) => {
  useEffect(() => {
    console.log(`${name} Layout mounted.`);
    console.log(
      `${name} Layout restProps:`,
      JSON.stringify(props, null, 2)
    );
    console.log(
      `${name} Layout arguments:`,
      JSON.stringify(args, null, 2)
    );
  }, []);
  console.log(`${name} Layout rendered.`);
  return (
    <div data-name={name} data-mode={mode}>
      {children}
    </div>
  );
};

export default Layout;

Create a pre-commit hook

Tue, 20 Aug 2019 12:25:00 GMT

A pre-commit hook preserves the quality of your code by running lint, format, and/or test scripts before commits. It prevents commits if it finds a problem in the code, or, in other words, if one of the previous scripts fails. A popular tool that is used for setting up pre-commit hooks is husky.

husky, is often used along with another tool the lint-staged. As the name suggests, lint-staged improves the performance of your pre-commit hook by allowing you to run scripts only against the staged files. For example, you don’t have to run the lint script for the entire project but only for the files that changed.

Install dependencies

Let’s see how you can create a pre-commit hook with husky and lint-staged. First, install the dependencies:

yarn add -D husky lint-staged

Configure husky

Then, configure husky to run the lint-staged script before you commit the code:

package.json

{
  "name": "pre-commit-hook",
  "private": true,
  "version": "1.0.0",
  "main": "src/index.js",
  "license": "MIT",
  "scripts": {},
  "husky": {
    "hooks": {
      "pre-commit": ["lint-staged"]
    }
  }
}

By the way, you can create more Git hooks with husky; you are not restricted to pre-commit hooks.

Linting

You now have to configure lint-staged. The following script runs ESLint against the staged files and fails—it won’t commit the code—if it finds a lint error:

package.json

{
  "name": "pre-commit-hook",
  "scripts": {
    "lint": "eslint --ext js,jsx src"
  },
  "husky": {
    "hooks": {
      "pre-commit": ["lint-staged"]
    }
  },
  "lint-staged": {
    "*.{js,jsx}": ["eslint"]
  }
}

Notice that you only run the eslint command in the lint-staged script and not the lint script I highlighted above. If you run the latter, it would run ESLint against the entire project multiple times—for each staged file— and would slow down the pre-commit hook without a reason. Needless to say that I’ve made that mistake in the past.

Formatting

In addition to linting, a pre-commit hook can also format the code. A popular formatting tool is Prettier. Depending on how you configured Prettier with ESLint, the format command may differ. In the following example, let’s assume that you configured ESLint to run Prettier (this is the best way), and as a result, you can run both tools with the eslint --fix command:

package.json

{
  "name": "pre-commit-hook",
  "husky": {
    "hooks": {
      "pre-commit": ["lint-staged"]
    }
  },
  "lint-staged": {
    "*.{js,jsx}": ["eslint --fix", "git add"]
  }
}

If you use prettier-eslint, you run prettier-eslint --write; if you use only Prettier, you run prettier --write.

After you format the code, you run git add to stage any JavaScript files that changed after the formatting.

From version 10 onwards, you don’t have to use git add to stage the files because lint-staged adds them automatically in the commit to prevent race conditions. In the next examples, I will omit it.

But because we can run ESLint only against our JavaScript files, the rest of our files—CSS, JSON, e.t.c.—remain unformatted. Let’s fix that by creating another file rule that runs prettier to format the remaining files:

package.json

{
  "name": "pre-commit-hook",
  "husky": {
    "hooks": {
      "pre-commit": ["lint-staged"]
    }
  },
  "lint-staged": {
    "*.{js,jsx}": ["eslint --fix"],
    "*.{md,mdx,json,css,scss,yaml,yml}": ["prettier --write"]
  }
}

See the lint-staged documentation for more examples.

Optional testing

Finally, you can run a test script— in this case with Jest—in your lint-staged script:

package.json

{
  "name": "pre-commit-hook",
  "husky": {
    "hooks": {
      "pre-commit": ["lint-staged"]
    }
  },
  "lint-staged": {
    "*.{js,jsx}": [
      "eslint --fix",
      "jest --findRelatedTests --bail"
    ],
    "*.{md,mdx,json,css,scss,yaml,yml}": ["prettier --write"]
  }
}

The --findRelatedTests flag instructs Jest to check if the staged files have tests and run only those tests. Additionally, with the --bail flag, Jest will stop testing if a single test fails.

Keep in mind, that most people prefer to run only lint and format scripts in their pre-commit hooks and leave the test scripts for Continuous Integration. But for simple projects, it’s ok to run test scripts too.

Unit testing with react-scripts

Mon, 19 Aug 2019 16:20:00 GMT

You can use react-scripts to write unit tests without setting up a build process with Babel and Webpack/Rollup. It’s useful when you want to quickly test an idea, but you’re not sure if you want to create a library yet. It can also be useful when your code lives inside a yarn workspace and doesn’t need a build process.

Install react-scripts

Install react-scripts as a devDependency by typing in your terminal:

yarn add -D react-scripts

Write test scripts

Create some test scripts in your package.json, that will help you run the tests:

package.json

{
  "scripts": {
    "test": "react-scripts test",
    "test:ci": "cross-env CI=true react-scripts test",
    "test:ci:alt": "react-scripts test --ci --watchAll=false",
    "test:jsdom": "yarn test --env=jsdom",
    "test:staged": "yarn test:ci --findRelatedTests",
    "test:coverage": "yarn test:ci --coverage"
  }
}

What the test scripts do

The test script runs the tests in watch mode—which is the default option.
If you want to run the tests and exit, you can set the CI environmental variable to true. That’s what the test:ci script does, with the help of the cross-env package. An alternative to that is to set the --watchAll flag to false.
If your tests require the DOM—if you are testing React components for example—you can use the test:jsdom script which sets the --env flag to jsdom.
The test:staged script is useful for running tests with a pre-commit hook—with husky and lint-staged for example.
Finally, the test:coverage script runs the tests and creates a coverage report. You can configure the coverage script further if you want.

Write tests

To write tests, use the test or it functions and assert your logic with the expect function. You can optionally organize your tests inside a describe function. In the following snippet, I wrote some tests for a “flatten” function, that takes a nested array as input and returns a flat array.

The tests:

src/flatten.test.js

import flatten from "./flatten";

describe("Flatten tests:", () => {
  it("Flattens an array.", () => {
    const nestedArray = [[1, 2, 3], [4]];
    const flatArray = [1, 2, 3, 4];
    expect(flatten(nestedArray)).toEqual(flatArray);
  });

  it("Returns empty array when the input is an empty array.", () => {
    const array = [];
    const result = flatten(array);
    const expectedResult = [];
    expect(result).toEqual(expectedResult);
  });
});

describe("Infinite nesting tests:", () => {
  it("Goes 4 levels deep.", () => {
    const nestedArray = [[1, 2, [3, [[4], 5]]], [6]];
    const flatArray = [1, 2, 3, 4, 5, 6];
    const result = flatten(nestedArray);
    expect(result).toEqual(flatArray);
  });
});

describe("Exact level tests:", () => {
  it("Goes exactly two levels deep.", () => {
    const nestedArray = [[1, [2, [3]], 4], [5]];
    const flatArray = [1, 2, [3], 4, 5];
    const result = flatten(nestedArray, 2);
    expect(result).toEqual(flatArray);
  });
});

The flatten function:

src/flatten.js

const flatten = (array, level = 9999) => {
  if (!Array.isArray(array)) return array;
  const shouldGoDeeper = (lvl) => lvl + 1 <= level;
  const toFlat = (lvl) => (result, item) => {
    if (shouldGoDeeper(lvl) && Array.isArray(item))
      return item.reduce(toFlat(lvl + 1), result);
    return result.concat(item);
  };
  return array.reduce(toFlat(1), []);
};

export default flatten;

Useful links

Useful VS Code shortcuts

Sun, 18 Aug 2019 12:10:00 GMT

In this post, you can find some practical VS Code shortcuts that can improve your productivity. They are shortcuts for navigating inside a file and between files, for text selection, and more. In the first section, you’ll see how you can change a shortcut.

Configure shortcuts

To change the default shortcuts, first, open the Go to File dropdown with Ctrl + P. Then, type the greater-than sign > to bring up the Command Palette. Finally, type "shortcuts", and select the Open Keyboard Shortcuts from the dropdown as shown in the video below.

How to Open Keyboard Shortcuts

You can open the command palette with Ctrl + Shift + P.

This will open Keyboard Shortcuts in a new open editor. From there, you can type the name of the shortcut you want to change or its key-binding. You can also search for keywords from within the command palette, without opening the keyboard shortcuts, but you can’t search for a shortcut by the keybinding. I find it useful to right-click to open the context menu and select what I want to change from the shortcut. For example, many keywords have a when clause. That means that you can activate them under specific conditions. In the next video, I change the default shortcut for Select All occurrences—notice that it has a when clause when the editor has focus.

How to change a shortcut

Select text

This section contains shortcuts that you can use to select text efficiently.

Select next/previous occurrence

Select some text and then press Ctrl + D to select the next occurrence of that text. In the video below, I use that shortcut to rename some React components.

Add next occurrence into the selection

You can use this shortcut to rename, for example, a <div> element to a <p> or remove some double-quotes. There is a shortcut to select the previous occurrence, but it’s not enabled by default. It’s called Add Selection To Previous Find Match. I configured it to Ctrl + Shift + D because it’s similar to the Ctrl + D of the next occurrence, so it’s easy to remember.

Select all occurrences

Select some text and then press Ctrl + Shift + L to select all occurrences of that text. It can be useful when you want to rename a component for example. I changed this shortcut to Ctrl + Shift + A because I associate the A with All.

If you want to avoid editing something accidentally, before you press Ctrl + Shift + L, press Ctrl + F instead and check how many times the text is present in the search pop up:

The number of occurrences as shown in the Search pop-up

If that doesn’t help, you can also press F3 to cycle through the occurrences to make sure you won’t overwrite something by accident. You can also get the number of selections from the status bar, at the bottom of your screen:

The number of selections as shown in the Status Bar

Smart select

I find kind of hard to explain what smart select does, so I suggest to see for yourself in the video below:

Smart select

Smart select can be useful when you want to select large blocks of code. In the example above, I select all the render code inside a React component. To do that, you place your cursor inside the code you want to select, and then you press Shift + Alt + Right Arrow to gradually expand the selection. If you messed it up and selected more than you wanted, press Shift + Alt + LeftArrow—or just LeftArrow since you are already holding Shift + Alt—to shrink the selection.

Smart Select doesn’t work well with JSX tags. By that I mean it doesn’t gradually select the parent tags but instead selects all the content inside the render method at once. For that reason, I use the Very Smart Select extension.

Undo cursor position

If you used select next/previous occurrence, and you selected more text than you wanted, you can press Ctrl + U to return to the previous cursor position. This also works with smart select or if you navigate between the top/bottom brackets that you’ll see in the navigation section. One problem with this shortcut, though, is that it’s close to the redo keyword Ctrl + Y. So before you press it, spend some seconds and try to recall the correct keybinding.

Select words

You can press Ctrl + Shift + Left/Right Arrow to select words. To move the cursor from word to word, you can press Ctrl + Left/Right Arrow.

Vertical selection

You can vertically select text with Ctrl + Alt + Down/Up Arrow. If you mess up the selection, press Escape and try again. You can also press the Ctrl + U shortcut you saw in the previous section to undo the cursor position.

A use case for the vertical selection shortcut is to copy a component’s props—in other words, the function parameters—and paste them in the line you want to render the component. Check the following video for an example:

Copy the props with vertical select:

Vertical select

Paste the selection in the render method. I’m also selecting the previous occurrence to replace the comma with ={""}:

Paste vertical selection

Undo/Redo

You probably know that you can undo recent changes with Ctrl + Z. But what if you go too far and discard some changes that you want? Press Ctrl + Y to redo the changes. This combination may not seem that impressive, but it’s really useful in practice.

This section contains shortcuts for navigating inside an open file, different files, or different views in VS Code. Instead of searching for the file in the explorer with your mouse, you can spend some seconds to think about what you want to accomplish, recall the shortcut, and use it to navigate fast without moving your hands away from the keyword.

Move to the start/end of a bracket.

Imagine that you are at the top bracket { of a long function, and you want to go fast to the bottom bracket }. In this case, you can break the function into smaller functions because it does more than it should. An alternative is to press Ctrl + Shift + \ (backslash) to toggle between the top and bottom brackets. I don’t use it frequently, but when I do, I find it really useful.

Top/bottom bracket navigation

Other ways to move quickly inside a file

Page Up and Page Down can help you move quickly when you are inside a file. Again, the undo cursor position Ctrl + U can be useful here.
To go to the top of the file, press Ctrl + Home, and to go to the bottom, press Ctrl + End.
For small distances, the Up/Down Arrow keys will be enough.
If you know what you’re looking for, you can search for the text with Ctrl + F and cycle through the occurrences with the F3 key. This is probably the fastest way to move in a long file.

Go to a specific line

If you want to go to a specific line—you maybe saw the number from an error call-stack—you can first press Ctrl + P to open the Go to File dropdown. Then, type the colon (:) character, after that the line you want to go to, and finally press Enter to jump to that line. An invaluable shortcut in the rare cases where you know the number of the line.

Instead of typing the colon character to navigate to a line, you can type the at sign @ to navigate to a symbol.

Move to a different file

If you know the name of the file, press Ctrl + P to open the Go to File dropdown and then type the name of the file. You can then select the file you want to open from the dropdown. If you type a generic file name like index.js, you may see a lot of results. You can limit the results if you know the exact file path, for example, you can instead type src/pages/index.js to get more relevant results.

How to navigate to a different file.

One dirty trick you can use after you open the file is to save it with Ctrl + S. You do that to pin the file in the open editors because if you try to open another file, you will replace it with the new file.

If you don’t know the name of the file, but you know what’s inside that file, you can press Ctrl + Shift + F to open the Search view. You can then type the text you’re looking for and open a file with the mouse. If you don’t want to use the mouse, you can press Tab to select the next interactive element until you reach the result list. To select a result, you’ll have to press the Down Arrow while the list is highlighted. If you press the Shift, you’ll probably end up in the file you have open (aka open editor).

Navigate to different views

In the previous section, I mentioned that you can use Ctrl + Shift + F to navigate to the Search view. It’s useful to know the shortcuts for the other views too. You can see those shortcuts inside the UI if you hover over the view icons with your mouse. The view icons are in the left-most sidebar.

The view icons in the sidebar

To open the Explorer view, press Ctrl + Shift + E.
To open the Source control view, press Ctrl + Shift + G, release the keys, and press G.
To open the Extensions view, press Ctrl + Shift + X.

These shortcuts are easy to remember because the last key corresponds to the name of the view. For example, E for the explorer view, G for git, and X for extensions. There is also the debug view that you can open with Ctrl + Shift + G, but, as I already mentioned a few million times, I use this keybinding for the Add Selection To Previous Find Match.

Shortcuts for terminals

You probably know that you can open and close the terminal if you press Ctrl + ` (the grave accent character also known as backtick). This shortcut is called View: Toggle Integrated Terminal. But there’s another shortcut that’s called: View: Focus Active Editor Group. If you configure the latter to the same keybinding—that is Ctrl + `—and you specify a when clause with a value of terminalFocus, you can then switch the focus between the terminal and the active editor. To add a when clause, right-click the setting and select change when expression.

You can move from terminal to terminal with Alt + Up/Down Arrow (if the terminal has focus), and you can split it (in other words, open a new terminal) with Ctrl + Shift + 5. Finally, to kill the active terminal, you need to configure the Terminal: Kill the Active Terminal Instance shortcut. I’ve configured mine to Ctrl + Shift + 6.

Delete a line

There are a lot of ways you can delete a line:

You can cut the text with Ctrl + X.
You can move the cursor to the line you want to delete and press Ctrl + Shift + K. Use this over the cut if you don’t want to override the text inside your clipboard. By cutting some text you also copy it in your clipboard.
You can select the entire line with Ctrl + L and then press Delete to delete the selection.
You can press Home to go at the beginning of the line, press Shift + End to select the line, and finally, press Delete to delete it.

Formatting

You can format an open document by pressing Shift + Alt + F.
You can toggle word wrap with Alt + Z which is useful when working with text files.

Pressing the `Alt` key alone

Many Windows shortcuts include the Alt key. For example, the Alt + Tab shortcut to switch between open windows. I use that shortcut all the time, but, sometimes, I might not press it correctly, and only the Alt registers. Pressing the Alt key alone in Windows programs transfers the focus to a menu bar, and you lose focus on what you’re working on. It’s good to know that when that happens inside VS Code, you can press the Alt key again to restore focus.

Resources

Fix oversized icons from react-fontawesome

Sat, 17 Aug 2019 12:10:00 GMT

Problem

You are using react-fontawesome to render some Fontawesome icons in your app, and you notice that in an SSR build your icons get huge on page load, only to shrink back to their normal size a bit later. This is what it looks like:

Bug demonstration

And this is the code:

/** @jsx jsx */
import { jsx, Box } from "theme-ui";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { faHeart } from "@fortawesome/free-solid-svg-icons";
import Layout from "@affectionatedoor/gatsby-theme-ui/src/components/Layout";

export default () => (
  <Layout>
    <Box
      sx={{
        position: "relative",
        minHeight: "70vh",
        bg: "mute",
        color: "accent",
      }}
    >
      <FontAwesomeIcon
        sx={{
          position: "absolute",
          top: "calc(50% - 20px)",
          left: "calc(50% - 20px)",
        }}
        icon={faHeart}
        size="2x"
      />
    </Box>
  </Layout>
);

Solution

To solve this—assuming you installed the @fortawesome/fontawesome-svg-core package—you’ll have to do 2 things:

Import the @fortawesome/fontawesome-svg-core/styles.css stylesheet before React loads. I’m using Gatsby, so I load the stylesheet in gatsby-browser.js:
gatsby-browser.js
```
import "@fortawesome/fontawesome-svg-core/styles.css";
```

Then prevent fontawesome from loading the CSS styles again:

import { jsx, Box } from "theme-ui";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { faHeart } from "@fortawesome/free-solid-svg-icons";
import Layout from "@affectionatedoor/gatsby-theme-ui/src/components/Layout";
import { config } from "@fortawesome/fontawesome-svg-core";

// You should do that in a Layout file or in `gatsby-browser.js`.
config.autoAddCss = false;

export default () => <Layout>{"Skipping content..."}</Layout>;

Now the icons are displayed in their correct size, as you can see in the following gif:

Bug resolved

Preserve new lines and spacing in CSS

Wed, 14 Aug 2019 13:20:00 GMT

Problem

You got some text (maybe from a server) that has newlines and spacing. This what the text looks like in JavaScript:

const text =
  "I have a newline here\n and some extra white           space for some reason.";

Now you want to display it in HTML, so you render it in JSX:

<p>{text}</p>

And this is the result:

I have a newline here and some extra white space for some reason.

The newlines and spaces are ignored.

Solution

If you want to preserve new lines and white space, you can use the white-space CSS property with the pre-wrap value:

p {
  white-space: pre-wrap;
}

So the JSX now becomes:

<p style={{ whiteSpace: "pre-wrap" }}>{text}</p>

And this is what it looks like:

I have a newline here
 and some extra white           space for some reason.

You can also use a preformatted text element <pre>.

How to break up long links with CSS

Wed, 14 Aug 2019 13:00:00 GMT

Problem

You have a really long link, for example, a link to a GitHub issue, and it overflows the container. This will usually happen in a mobile layout:

https://github.com/facebook/react/issues/14920#issuecomment-471070149

This is the code for the link:

<a
  href="https://github.com/facebook/react/issues/14920#issuecomment-471070149"
  style={{ color: "#f18fe8" }}
>
  https://github.com/facebook/react/issues/14920#issuecomment-471070149
</a>

This can happen if you create a link in Markdown and you forget to give it a title:

## Useful links

- [https://github.com/facebook/react/issues/14920#issuecomment-471070149](https://github.com/facebook/react/issues/14920#issuecomment-471070149)

Solution

To solve this, either write a descriptive text for the link—the best solution—or use one of the following CSS properties:

And if you do that, this is the result:

https://github.com/facebook/react/issues/14920#issuecomment-471070149

<a
  href="https://github.com/facebook/react/issues/14920#issuecomment-471070149"
  style={{
    color: "#f18fe8",
    wordBreak: "break-word",
  }}
>
  https://github.com/facebook/react/issues/14920#issuecomment-471070149
</a>

Dark mode in React with Theme UI

Tue, 13 Aug 2019 01:00:00 GMT

A few months ago, I searched for the best dark mode solution for a React application. I wrote a post about that if you are interested. But that was three months ago, and three months is a long time in web development—not really, just trying to create some drama. Nowadays, the solution to dark mode is a lot easier and has a name: Theme UI.

With the Theme UI library, you can style your applications consistently with a design system (styled-system) and a CSS-in-JS library (emotion). Additionally, Theme UI offers support for MDX, and who doesn’t love MDX? In the following examples, I will use Gatsby because the MDX support is great. But you can use MDX with many other React apps; you don’t have to use Gatsby.

Let’s see now in practice how you can add dark mode to your site with Theme UI.

The easy way

GitHub repository. Checkout to the with-custom-theme branch.

The easy way to add Theme UI in your application is to use a Gatsby theme I made. There’s nothing special about that theme; it just setups Theme UI and exposes some UI components I use in my projects. Those components include background images, animation on scroll components, buttons, and many others.

But before you can enjoy dark mode in your application, you’ll have to do 4 things:

Create a new hello-world starter:

gatsby new . git@github.com:gatsbyjs/gatsby-starter-hello-world.git

Install my theme:

yarn add @affectionatedoor/gatsby-theme-ui

Add it as a plugin in gatsby-config.js:

gatsby-config.js

module.exports = {
    plugins: ["@affectionatedoor/gatsby-theme-ui"]
};

Use the Layout component from my theme in the index page:

src/pages/index.js

import React from "react";
import Layout from "@affectionatedoor/gatsby-theme-ui/src/components/Layout";

export default () => <Layout>Hello world!</Layout>;

If you start the development server and visit the homepage, you will see this screen:

Homepage with my Gatsby theme

You can also try to build and serve the app locally with yarn build and yarn serve. You will not experience any of the problems with the context that I mentioned in my previous post. Theme UI implements the last solution I showed with the CSS custom properties, the same solution Dan Abramov uses for his personal blog. Not by default, but with a small option—you’ll see that in a bit.

But you may not need all the components I have stuffed inside that theme. You may just don’t want to use my stinky theme. That’s fine, I don’t blame you. Instead, you may want something less opinionated. If that’s the case, you can use the official Gatsby Theme UI theme.

Using the official theme

GitHub repository - master branch.

The official Theme UI theme for Gatsby, let’s refer to it as the official theme, doesn’t do a lot of things; it’s the definition of minimal. It wraps Gatsby’s root element with a ThemeProvider, and it creates some empty files for the theme and the custom MDX components (don’t worry about the last, they are not important). We can then shadow those empty files in our app, and fill them with our code.

But before we do that, let’s install the dependencies first:

yarn add theme-ui gatsby-plugin-theme-ui @emotion/core @mdx-js/react

Then, add the plugin in the plugin array:

gatsby-config.js

module.exports = {
  plugins: ["gatsby-plugin-theme-ui"],
};

The next thing you’ll have to do is overwrite, or shadow, the empty theme file from the official theme:

src/gatsby-plugin-theme-ui/index.js

export default {
  useCustomProperties: true,
  initialColorMode: "light",
  colors: {
    text: "#000",
    background: "#fff",
    primary: "#07c",
    modes: {
      dark: {
        text: "#fff",
        background: "#000",
        primary: "#0cf",
      },
    },
  },
};

You can add a ton of useful stuff inside the theme object, but here I add only the colors, in an attempt to be minimal like the official theme. Also, notice the useCustomProperties field I highlighted in the code above. I said before that it’s a good idea to use CSS custom properties. If you don’t use them, and you change the theme from light to dark in production, when you reload the page, you’ll see a brief flash. That flash is caused because you see momentarily the light theme before the dark—your chosen theme in this case. It has to do with how React’s hydrate method works.

Now that you have a basic theme, it’s time to use it in your code. Open the index page, and paste the following code:

src/pages/index.js

/** @jsx jsx */
import { jsx, Styled } from "theme-ui";

import ThemeSwitcher from "../components/theme-switcher";

export default () => {
  return (
    <div
      sx={{
        backgroundColor: "background",
        color: "text",
        minHeight: "100vh",
        display: "flex",
        alignItems: "center",
        justifyContent: "center",
        flexDirection: "column",
      }}
    >
      <ThemeSwitcher />
      <Styled.h1>Hello world!</Styled.h1>
    </div>
  );
};

If you are not familiar with emotion, you may be wondering what the @jsx jsx comment does at the top of the file. With that comment, you declare that you want to use the custom JSX pragma from Theme UI. What this thing does is not as scary as it sounds. Let me explain.

Your JSX code at build time is transformed to React.createElement function calls. For example, this div:

<div style={{ color: "red" }}>Hello World</div>

During build time becomes this:

React.createElement(
  "div",
  {
    style: {
      color: "red",
    },
  },
  "Hello World"
);

If you don’t believe me, you can watch this process real-time with the Babel Online compiler. The createElement function takes 3 arguments: the component type, the props, and the children.

So, coming back to the custom pragma, instead of using the React.createElement function, we use the jsx function from Theme UI. For example, the React.createElement you saw in the previous code snippet becomes:

import { jsx } from "theme-ui";
jsx(
  "div",
  {
    style: {
      color: "red",
    },
  },
  "Hello World"
);

We do that because we want to use the sx prop from Theme UI—which is not a standard JSX property like the style for example—to style our components. Speaking of the sx prop, I use it on the index page to position the content of the div at the center of the screen. I also use a ThemeSwitcher component that doesn’t exist yet. This component is a button that switches our themes, as the name suggests.

Let’s now create that ThemeSwitcher component. Create a new file at src/components/theme-switcher.js and paste the following code:

src/components/theme-switcher.js

/** @jsx jsx */
import { jsx, useColorMode } from "theme-ui";

const ThemeSwitcher = () => {
  const [colorMode, setColorMode] = useColorMode();
  const nextColorMode = colorMode === "light" ? "dark" : "light";
  return (
    <button
      sx={{
        position: "absolute",
        top: 3,
        right: 3,
        backgroundColor: "primary",
        color: "background",
        border: 0,
        px: 3,
        py: 2,
        cursor: "pointer",
      }}
      onClick={() => setColorMode(nextColorMode)}
    >
      Toggle mode
    </button>
  );
};

export default ThemeSwitcher;

In the ThemeSwitcher component, I use the sx prop to style the button because no one likes ugly buttons, and I also use the useColorMode hook that allows us to change the theme.

And that’s it. If you now start the development server, this is what you’ll see:

Homepage with the official theme

Wrapping up, I want to say that few times I feel joy using a JavaScript library. When I use Theme UI, it’s one of those times (lol).

Modals in Gatsby

Thu, 25 Jul 2019 09:13:00 GMT

Imagine that you want to build an Instagram-like app in Gatsby. You display a grid of images on the homepage, and when you click on an image, you render it inside a modal. That modal has a full-size version of the image and some extra information. The information may be the text of the post or counts for likes and comments. If you are a visual person, this is what the finished app looks like.

The modal is an important part of the application, so you want it to satisfy the following requirements:

The modal should be accessible and somewhat pretty. A popular library that can help you with that is react-modal.
You also want to have good SEO. To achieve that, you want to display the modal on a different page. By rendering the modal on a different page, Google is now able to index the extra post information. Another benefit is that you can share the picture with a link. Finally, you can use the back button to close the modal and return to the homepage. The last is something the users expect. If you don’t do that, you risk your users leaving the page while trying to close the modal (lol).

So how can you implement that modal in Gatsby?

Create the pages

First, you have to create the pages from your data. In this example (repo link), I’m using the gatsby-source-instagram plugin as my data source. I’m querying the last 12 posts from alicexz’s Instagram account. I’m also using a theme that setups theme-ui in Gatsby and exposes some components. This is what my gatsby-config.js looks like:

gatsby-config.js

module.exports = {
  siteMetadata: {
    title: "Alice X. Zhang",
  },
  plugins: [
    "@affectionatedoor/gatsby-theme-ui",
    {
      resolve: `gatsby-source-instagram`,
      options: {
        username: `alicexz`,
      },
    },
  ],
};

I create the pages with the createPages API from Gatsby Node:

gatsby-node.js

exports.createPages = async ({ graphql, actions: { createPage } }) => {
  const result = await graphql(`
    query {
      allInstaNode {
        nodes {
          id
        }
      }
    }
  `);

  if (result.errors) {
    console.log(result.errors);
    return;
  }

  const postTemplate = require.resolve(
    "./src/templates/InstaModalPost/index.jsx"
  );
  result.data.allInstaNode.nodes.forEach((post) => {
    createPage({
      path: post.id,
      component: postTemplate,
      context: {
        id: post.id,
      },
    });
  });
};

Now that we have the pages, the next thing you have to do is to render the modal.

A good place to render the modal is in the post template. But now the question is, how can you render the index page that’s sitting behind the modal?

If you take a look at gatsbygram (an example app that literally does the same thing we do here), you can see that it uses a PageRenderer component to render the background page. At the time of writing, PageRenderer is an undocumented component. This means that is not supposed to be used that much (?). You can see that the only prop that’s being passed to the PageRenderer component is the location. This will be useful later.

You’re now going to do something similar in the post template file. In gatsbygram, this is done in a Layout component, but I find it easier to do in the template file. The previous link has the completed code. If you want to follow along, better paste the code below for now:

src/templates/InstaModalPost/index.jsx

/** @jsx jsx */
import { useState } from "react";
import PropTypes from "prop-types";
import { jsx, Styled } from "theme-ui";
import { graphql, navigate, PageRenderer } from "gatsby";
import Img from "gatsby-image";
import Modal from "react-modal";
import Link from "@affectionatedoor/gatsby-theme-ui/src/components/Link";

import { Times, Heart, Comment } from "../../components/Icons";

import "./style.css";

Modal.setAppElement(`#___gatsby`);

const modalStyles = {
  overlay: {
    backgroundColor: "rgba(0, 0, 0, 0.58)",
  },
  content: {
    position: "relative",
    top: "auto",
    left: "auto",
    right: "auto",
    bottom: "auto",
    maxWidth: "960px",
    margin: "32px auto",
    padding: 0,
    border: 0,
  },
};

const InstaPostTemplate = ({
  data: {
    instaNode: {
      caption,
      comments,
      id,
      likes,
      localFile: {
        childImageSharp: { fluid },
      },
    },
  },
}) => {
  
  const [modalOpen, setModalOpen] = useState(true);
  const modalCloseTimeout = 300;
  const closeModal = () => {
    setModalOpen(false);
    setTimeout(() => navigate(`/`), modalCloseTimeout);
  };
  return (
    <div>
      <PageRenderer key={"/"} location={{ pathname: "/" }} />
      <Modal
        isOpen={modalOpen}
        onRequestClose={closeModal}
        style={modalStyles}
        contentLabel="Modal"
        closeTimeoutMS={modalCloseTimeout}
      >
        <Styled.root>
          <div
            sx={{
              bg: "background",
              display: "flex",
              flexDirection: "column",
              position: "relative",
            }}
          >
            <Link
              to="/"
              aria-label="close modal"
              sx={{
                width: "50px",
                height: "50px",
                position: "absolute",
                display: "flex",
                alignItems: "center",
                justifyContent: "center",
                m: 3,
                right: 0,
                top: 0,
                zIndex: 1,
                color: "white",
                borderRadius: "50%",
                ":hover": {
                  color: "white",
                  bg: "#fbf9f926",
                },
              }}
              onClick={(e) => {
                e.preventDefault();
                closeModal();
              }}
            >
              <Times sx={{ fontSize: 4 }} />
            </Link>
            <Img title={id} fluid={fluid} />
            <div sx={{ mt: 3, px: [3, 4] }}>
              <small sx={{ mr: 3 }}>
                <Heart sx={{ color: "secondary", fontSize: 3 }} />{" "}
                {likes}
              </small>
              <small sx={{ mr: 3 }}>
                <Comment sx={{ color: "secondary", fontSize: 3 }} />{" "}
                {comments}
              </small>
            </div>
            <p sx={{ my: 4, px: [3, 4] }}>{caption}</p>
          </div>
        </Styled.root>
      </Modal>
    </div>
  );
};

InstaPostTemplate.propTypes = {
  data: PropTypes.shape({
    instaNode: PropTypes.shape({
      caption: PropTypes.string.isRequired,
      comments: PropTypes.number.isRequired,
      id: PropTypes.string.isRequired,
      likes: PropTypes.number.isRequired,
      localFile: PropTypes.shape({
        childImageSharp: PropTypes.shape({
          fluid: PropTypes.object.isRequired,
        }).isRequired,
      }).isRequired,
    }).isRequired,
  }).isRequired,
};

export default InstaPostTemplate;

export const query = graphql`
  query InstaPostModal($id: String) {
    instaNode(id: { eq: $id }) {
      likes
      id
      localFile {
        childImageSharp {
          fluid(maxWidth: 1200) {
            ...GatsbyImageSharpFluid
          }
        }
      }
      comments
      caption
    }
  }
`;

src/templates/InstaModalPost/style.css

/* react-modal transitions */
.ReactModal__Overlay {
  overflow: auto;
  opacity: 0;
  transform: translateY(-10%);
  transition: opacity 0.3s cubic-bezier(0.445, 0.05, 0.55, 0.95), transform
      0.3s cubic-bezier(0.445, 0.05, 0.55, 0.95);
}
.ReactModal__Overlay--after-open {
  opacity: 1;
  transform: translateY(0);
}

.ReactModal__Overlay--before-close {
  opacity: 0;
  transform: translateY(-10%);
}

If we run the code in development (after you create the index page, the other 2 components and shadow the layout from the theme), everything works just fine:

Artwork by Alice X. Zhang

But if you try to build the project, you see that the background page is missing:

Artwork by Alice X. Zhang

PageRenderer problem

This happens because the PageRenderer behaves differently in production. If you take a look at the development version of page renderer, you will see that the component downloads the page data using the location prop. This does not happen in production. Following the changes in the production file, I suspect that this happened recently as a part of some performance optimizations.

As a result, if you want to use the PageRenderer component to display the index page behind the modal, you’ll have to download the page data yourself. In the live gatsbygram site, the modal works fine without downloading the page data. But that happens because the deployed version of the site probably uses an earlier version of Gatsby.

But how to get the page data?

Getting the page data

All the browser API’s that you can implement in your gatsby-browser.js file, take the loadPage method as a parameter. This method accepts a page path and returns the data for that page. This is exactly what we want. So the plan now is to implement an appropriate browser API where:

We download the index page data,
and store them somewhere to use later in the PageRenderer component.

A good candidate for that is to implement the onPrefetchPathname method. This method is called whenever Gatsby prefetches the data for a page at idle time. This happens when it encounters a Link for that page. For example, if you have in the index page a Link to page-2, Gatsby will download the data for page-2 at idle time. This way, the navigation between pages feels instant. For this to work, you’ll have to add a link to the homepage in the template. In our case, the link is already in the template. It’s the button that closes the modal. This is the implementation of the onPrefetchPathname method:

gatsby-browser.js

import "@fortawesome/fontawesome-svg-core/styles.css";

// This only works if in the modal page you add a
// Link to the homepage.
export const onPrefetchPathname = ({ loadPage }) => {
  if (window.indexPageData === undefined) {
    loadPage("/").then((result) => {
      window.indexPageData = result;
      
      
      
      if (window.setIndexPageData) window.setIndexPageData();
    });
  }
};

// https://github.com/gatsbyjs/gatsby/issues/7454#issuecomment-425403812
export const shouldUpdateScroll = ({
  routerProps: { location },
  getSavedScrollPosition,
}) => {
  return getSavedScrollPosition(location);
};

You can see that I’m saving the page data in the window. I’m also calling the setIndexPageData method, if it’s truthy, that causes a re-render by setting the state in the template. If we load the modal page directly, the initial result is null for some reason. This is what you have to add in the template file:

src/templates/InstaModalPost/index.jsx

import { useState, useEffect } from "react";

// skipping imports and modal styles...

const InstaPostTemplate = (
  {
    // skipping props..
  }
) => {
  
  const building = typeof window === "undefined";
  const [indexPageData, setIndexPageData] = useState(
    !building && window.indexPageData
  );
  useEffect(() => {
    window.setIndexPageData = () => {
      setIndexPageData(window.indexPageData);
    };
  }, []);

  // Modal stuff.
  const [modalOpen, setModalOpen] = useState(true);
  const modalCloseTimeout = 300;
  const closeModal = () => {
    setModalOpen(false);
    setTimeout(() => navigate(`/`), modalCloseTimeout);
  };
  return (
    <div>
      <PageRenderer
        key={"/"}
        location={{ pathname: "/" }}
        pageResources={indexPageData}
        path="/"
      />
      {/* skipping... */}
    </div>
  );
};

One thing to note about prefetching data is that it may not always work. For example, the mobile Chrome offers the “Lite mode” where the browser refuses to download data for pages the user may never visit. In this case, we are not able to render the background page by implementing the onPrefetchPathname method.

Scroll position caveat

You may also notice in the previous gatsby-browser.js snippet, that I’m implementing the shouldUpdateScroll API. If you don’t do that, and you return to the index page after closing the modal, you’ll lose your scroll position in the index page, and you’ll start at the top of the page. This is obviously really annoying, and the implementation above fixes it.

Further improvement

The modal in this example could be further improved. For example, we could add next and previous buttons to cycle through the images when the modal is open. Gatsbygram does this but also adds keyboard navigation with the mousetrap library. There are some accessibility issues as well. For example, the modal pages do not have a Heading 1 element because Instagram API does not have a title for posts. Finally, because we’re downloading data for many pages (when the modal opens), the performance may suffer as well.

Change the URLs of posts in Gatsby

Wed, 17 Jul 2019 17:00:00 GMT

If you want to create an MDX blog in Gatsby, you have to do 3 things:

Install 3 packages: gatsby-plugin-mdx, @mdx-js/mdx, and @mdx-js/react.
Add the gatsby-plugin-mdx to your plugins array inside gatsby-config.js.
Create some MDX files inside src/pages.

And now you have an MDX blog. If you create a src/pages/post-1.mdx file, Gatsby will create a page for that file and will be available at:

http://example.com/post-1/

But sometimes, you want your posts to be under a /posts/ URL. For example:

http://example.com/posts/post-1/

One way to do this is to create the MDX pages yourself with the createPages API. But creating the pages this way requires 3 extra steps:

You have to implement the createPages API in your gatsby-node.js to create the pages and maybe the onCreateNode to add extra fields to your nodes—for example, a slug in the Mdx GraphQL type.
You have to install and configure the gatsby-source-filesystem plugin to look for MDX files. This is important because otherwise, your MDX files won’t be available to GraphQL queries; and you want that for the first step (to create the pages with the createPages API).
Finally, you have to create a post template to use it as a layout for your post pages.

But that’s only one way to create custom URLs for your posts; let’s now see what alternatives you have.

Folder solution

If you want to change only the URL of your posts, you can place your MDX files inside folders. For example, if you want a bunch of notes under the /notes/ URL, create a notes folder, and place your notes inside that folder. Consider the following file structure:

src
|__ pages
|____ index.js
|____ page-2.js
|____ notes
|______ note-1.mdx
|______ note-2.mdx

This solution has some limitations though. For example, you can’t pass context data to your page or use a custom template for the posts. If you want to have those options, you can use the onCreatePage API.

onCreatePage API

You can use the createPages API to programmatically create pages. But in addition to the createPages, there is another Node API: the onCreatePage. This API is called, as you may have guessed, when a page is created. But an important detail here is that is not called for those created by the createPages.

So the plan is to use the onCreatePage to delete the created pages from src/pages and then create new pages with the updated path. You don’t want to delete all the pages, though, only the MDX pages that you’ll use as posts. To identify them, you can add a frontmatter field called type with a value of post at the top of the file. For example:

---
type: post
---

# Post 1

Content

onCreatePage example

To see this in action, create a new project with the hello-world starter:

mkdir change-post-url
cd change-post-url
gatsby new . https://github.com/gatsbyjs/gatsby-starter-hello-world.git

Install gatsby-plugin-mdx and the other dependencies:

yarn add gatsby-plugin-mdx @mdx-js/mdx @mdx-js/react

Configure the plugin in your gatsby-config.js:

gatsby-config.js

module.exports = {
  plugins: ["gatsby-plugin-mdx"],
};

Then, create an MDX file inside src/pages that will be used as a regular page:

src/pages/page-2.mdx

# Page 2

This is the second page in MDX.

import { Link } from "gatsby"

<Link to="/">Go back to homepage</Link>

And another MDX file that you’ll use as a post:

src/pages/post-1.mdx

---
type: post
---

# Post 1

This is my first post.

After you do all the above, you end up with 3 pages:

/
/page-2/
/post-1/

What’s left now is to change the URL of /post-1/ page to /posts/post-1/. Implement the onCreatePage API in your gatsby-node.js:

gatsby-node.js

exports.onCreatePage = ({
  page,
  actions: { createPage, deletePage },
}) => {
  const frontmatter = page.context.frontmatter;
  if (frontmatter && frontmatter.type === "post") {
    deletePage(page);
    createPage({
      ...page,
      path: `/posts${page.path}`,
    });
  }
};

If the page has a frontmatter object in the context, then we’re dealing with an MDX page. To check if it’s also a post, you look inside that frontmatter object for a type field with a value of "post". If all the previous are true, you delete the page, and create a new one with the updated path:

Notice that I change only the path in the createPage method. You can also pass some context or a template (via the component field) if you want. Check the createPage method for more details.

After implementing the onCreatePage API, you end up with the following pages:

/
/page-2/
/posts/post-1/

You can do that for any data you display on your site; it doesn’t have to be a post. If you’re building a hotel website, that data could be rooms or special offers.

Get the MDX files with GraphQL

You may want to create a page that lists all the posts. To do that, you want to get your MDX pages with a GraphQL query. gatsby-source-filesystem can help you with that. First, install the plugin:

yarn add gatsby-source-filesystem

And configure it in your gatsby-config.js to look for files in your src/pages:

gatsby-config.js

module.exports = {
  plugins: [
    "gatsby-plugin-mdx",
    {
      resolve: "gatsby-source-filesystem",
      options: {
        name: "mdx",
        path: `${__dirname}/src/pages`,
      },
    },
  ],
};

List all posts example

If you want to show only the posts (type: post), you can write a GraphQL query like the following:

`
AllPostsQuery {
  allMdx(filter: { frontmatter: { type: { eq: "post" } } }) {
    nodes {
      id
      timeToRead
      excerpt(pruneLength: 200)
    }
  }
}
`;

The following snippet shows how to display the posts in the index page:

src/pages/index.js

import React from "react";
import { graphql, Link } from "gatsby";



import slugify from "slugify";

export default ({
  
  data: {
    allMdx: { nodes: posts },
  },
}) =>
  console.log(posts) || (
    <main>
      <h1>Hello world!</h1>
      <p>In the next section, you can find all the posts.</p>
      <h2>Posts</h2>
      <ul>
        {posts.map((post) => (
          <li key={post.id}>
            <Link
              to={`/posts/${slugify(post.headings[0].value, {
                lower: true,
              })}/`}
            >
              <h3>{post.headings[0].value}</h3>
            </Link>
            <p>{post.excerpt}</p>
          </li>
        ))}
      </ul>
    </main>
  );

export const query = graphql`
  {
    allMdx(filter: { frontmatter: { type: { eq: "post" } } }) {
      nodes {
        id
        timeToRead
        excerpt(pruneLength: 200)
        headings(depth: h1) {
          value
          depth
        }
      }
    }
  }
`;

There are better ways to get the slug of the page; don’t consider what I did above a good practice. I use slugify to transform the <h1> that I get from GraphQL—assuming that you should only have one <h1> per page. A better way would be to add the slug yourself as a frontmatter field in the post. You can also do it programmatically on built time with onCreateNode, and make it available in your mdx GraphQL queries as a fields property—this is a common practice but adds more complexity.

Use the page creator plugin

Another thing you may want to do is to move your MDX posts from the src/pages directory to a posts directory. You can use the gatsby-plugin-page-creator to create pages from a different folder. First, install the plugin:

yarn add gatsby-plugin-page-creator

Then, configure it in your gatsby-config.js to create pages from the files in the posts folder (also change the folder for the gatsby-source-filesystem):

gatsby-config.js

module.exports = {
  plugins: [
    "gatsby-plugin-mdx",
    {
      resolve: "gatsby-source-filesystem",
      options: {
        name: "posts",
        path: `${__dirname}/posts`,
      },
    },
    {
      resolve: `gatsby-plugin-page-creator`,
      options: {
        path: `${__dirname}/posts`,
      },
    },
  ],
};

If you move the post-1.mdx file from src/pages to posts, everything should work as before.

Working with data in Gatsby

Sun, 23 Jun 2019 15:12:03 GMT

In this post, you will see how to work with data in Gatsby from hard coding them, to transformer plugins, and finally to MDX.

Say you want to create an application for a travel agency. The agency organizes 6 tours in different countries over the world. Their homepage should have a section that lists those tours. They also want to show on a different page some extra information about each tour. The first thought that comes to my mind is to create a section on the homepage where each tour will be represented by a card. That section may look like this:

The cards in tours section

What’s the best way to implement this requirement?

By the way, I created a repo on GitHub if you want to see the complete code. I'm using styled-system and rebass to style things.

First of all, you have to create the Card component (you can find the code in the link). The card component needs 5 props:

A title.
A description.
A price.
A URL that links to the “read more” page.
An image.

The first solution is to create in JSX 6 different cards and hard code the data for each tour.

Hard coding the data

To do this, go to the index page, and create 6 different cards. In each card, you pass down the text data as props.

You also want to pass an image. For that, put your images in a folder named src/images/tours, and write a GraphQL query to get them.

The full code for the index page is the following:

src/pages/index.jsx

import React from "react";
import { graphql } from "gatsby";

import { Flex, Box } from "../components/Primitives";
import { Heading1 } from "../components/Variants";
import Layout from "../layouts/Layout";
import SEO from "../components/SEO";
import TourCard from "../components/TourCard";

// eslint-disable-next-line react/prop-types
const IndexPage = ({ data }) => {
  const tours = data.allFile.edges.map(({ node }) => ({
    image: node.childImageSharp.fluid,
    originalName: node.childImageSharp.fluid.originalName,
  }));
  return (
    <Layout>
      <SEO title="Home" keywords={["gatsby", "application", "react"]} />
      <Box height="calc(100vh - 100px)" pt={5}>
        <Heading1 textAlign="center" mb={5}>
          Available Tours
        </Heading1>
        <Flex
          flexWrap="wrap"
          style={{ maxWidth: "1200px", margin: "0 auto" }}
        >
          <TourCard
            title="Spain"
            fluidImage={
              tours.find(({ originalName }) =>
                originalName.match(/spain/i)
              ).image
            }
            price="800$"
            description="1 Week in Spain"
            url="/tours/spain"
          />
          <TourCard
            title="France"
            fluidImage={
              tours.find(({ originalName }) =>
                originalName.match(/France/i)
              ).image
            }
            price="800$"
            description="1 Week in France"
            url="/tours/france"
          />
          <TourCard
            title="Germany"
            fluidImage={
              tours.find(({ originalName }) =>
                originalName.match(/Germany/i)
              ).image
            }
            price="800$"
            description="1 Week in Germany"
            url="/tours/germany"
          />
          <TourCard
            title="Greece"
            fluidImage={
              tours.find(({ originalName }) =>
                originalName.match(/Greece/i)
              ).image
            }
            price="800$"
            description="1 Week in Greece"
            url="/tours/greece"
          />
          <TourCard
            title="Japan"
            fluidImage={
              tours.find(({ originalName }) =>
                originalName.match(/Japan/i)
              ).image
            }
            price="800$"
            description="1 Week in Japan"
            url="/tours/japan"
          />
          <TourCard
            title="USA"
            price="800$"
            description="1 Week in USA"
            url="/tours/usa"
            fluidImage={
              tours.find(({ originalName }) => {
                return originalName.match(/USA/i);
              }).image
            }
          />
        </Flex>
      </Box>
    </Layout>
  );
};

export const query = graphql`
  query TourQuery {
    allFile(filter: { relativePath: { regex: "/tours/" } }) {
      edges {
        node {
          childImageSharp {
            fluid {
              ...GatsbyImageSharpFluid_tracedSVG
              originalName
            }
          }
        }
      }
    }
  }
`;

export default IndexPage;

There is a minor problem with this method. The text data are not linked with the image. That means that you have to write a function to compare the originalName of the image with the current card name, as shown below:

src/pages/index.jsx

<TourCard
  title="Japan"
  fluidImage={
    tours.find(({ originalName }) => originalName.match(/Japan/i)).image
  }
  price="800$"
  description="1 Week in Japan"
  url="/tours/japan"
/>

The other requirement is to create pages with additional information for each tour. For that, you can either create custom pages in src/pages with JSX or create a blog with markdown or MDX. That’s true for every method you’ll see here, and we’ll ignore that requirement for now.

Hard coding the tour data is fine if you don’t have a lot of items. In our case, the items are 6 which is not a small number. If the agency increases the tours in the future, it will become unmanageable.

Before we see the next method, let’s make an improvement on the current one.

Hard coding improvement

If you want to avoid hard coding and writing 6 different cards, you can create a JSON or a JavaScript file that contains the tour data. You can then import that file, and map through it to display the tours.

Create a file with the tour data in src/data/tours.js:

src/data/tours.js

export default [
  {
    title: "Spain",
    price: "800$",
    description: "1 Week in Spain",
    url: "/tours/spain",
  },
  {
    title: "France",
    price: "800$",
    description: "1 Week in France",
    url: "/tours/france",
  },
  {
    title: "Germany",
    price: "800$",
    description: "1 Week in Germany",
    url: "/tours/germany",
  },
  {
    title: "Greece",
    price: "800$",
    description: "1 Week in Greece",
    url: "/tours/greece",
  },
  {
    title: "Japan",
    price: "800$",
    description: "1 Week in Japan",
    url: "/tours/japan",
  },
  {
    title: "USA",
    price: "800$",
    description: "1 Week in USA",
    url: "/tours/usa",
  },
];

Then, import that file in src/pages/index.js, and use it:

src/pages/index.jsx

// skipping
<Flex flexWrap="wrap" style={{ maxWidth: "1200px", margin: "0 auto" }}>
  {tourData.map(({ title, price, description, url }) => {
    const regExp = new RegExp(title, "i");
    return (
      <TourCard
        key={title}
        title={title}
        fluidImage={
          tours.find(({ originalName }) => originalName.match(regExp))
            .image
        }
        price={price}
        description={description}
        url={url}
      />
    );
  })}
</Flex>
// skipping

The code above is better because we avoid duplication. But we still have the problem where the images are not linked with the rest of the data. To solve this, you can use a “transformer” plugin.

gatsby-transformer-json

Another option you have is to take advantage of the Gatsby source plugins. You’ll do almost the same thing you did in the hard coding improvement.

You will place your data in a JSON file, and the gatsby-transformer-json plugin will transform them into “tour” objects. Then, you will write a GraphQL query to get those objects.

If you prefer the YAML format over JSON, you can use the YAML plugin.

You start by installing the plugin:

yarn add gatsby-transformer-json

Then, you configure it in your gatsby-config.js:

gatsby-config.js

const pkg = require("./package");

module.exports = {
  siteMetadata: {
    title: pkg.name,
    description: pkg.description,
    author: pkg.author,
  },
  plugins: [
    "gatsby-plugin-react-helmet",
    "gatsby-transformer-json",
    {
      resolve: "gatsby-source-filesystem",
      options: {
        path: `${__dirname}/src/data`,
      },
    },
    {
      resolve: "gatsby-source-filesystem",
      options: {
        name: "images",
        path: `${__dirname}/src/images`,
      },
    },
    // skipping
  ],
};

The next thing you have to do is to you create a file in src/data/tours.json with the following content. Remove the JavaScript file you added earlier, or transform it into a JSON file:

Notice the highlighted image fields:

src/data/tours.json

[
  {
    "title": "Spain",
    "price": "800$",
    "description": "1 Week in Spain",
    "url": "/tours/spain",
    "image": "images/tours/spain.jpg"
  },
  {
    "title": "France",
    "price": "800$",
    "description": "1 Week in France",
    "url": "/tours/france",
    "image": "images/tours/france.jpg"
  },
  {
    "title": "Germany",
    "price": "800$",
    "description": "1 Week in Germany",
    "url": "/tours/germany",
    "image": "images/tours/germany.jpg"
  },
  {
    "title": "Greece",
    "price": "800$",
    "description": "1 Week in Greece",
    "url": "/tours/greece",
    "image": "images/tours/greece.jpg"
  },
  {
    "title": "Japan",
    "price": "800$",
    "description": "1 Week in Japan",
    "url": "/tours/japan",
    "image": "images/tours/japan.jpg"
  },
  {
    "title": "USA",
    "price": "800$",
    "description": "1 Week in USA",
    "url": "/tours/usa",
    "image": "images/tours/usa.jpg"
  }
]

With that image field, you can now place your tour images in the src/data/images/tours folder. They will be available in GraphQL queries as childImageSharp fields inside the tour objects. You no longer have to search for the images.

The gatsby-transformer-json plugin created for us 2 GraphQL types: The allToursJson and toursJson.

Because we want all the tours, write in your index page a GraphQL query that uses the allToursJson:

export const query = graphql`
  query TourQuery {
    allToursJson {
      edges {
        node {
          description
          price
          title
          url
          image {
            childImageSharp {
              fluid {
                ...GatsbyImageSharpFluid_tracedSVG
              }
            }
          }
        }
      }
    }
  }
`;

With this query in place, your tour data will be available in the data.allToursJson field. You now have to go to the index page, and display the result:

src/pages/index.jsx

// skipping imports

const IndexPage = ({ data }) => {
  const tours = data.allToursJson.edges.map(({ node }) => {
    const { description, price, title, url, image } = node;
    return {
      description,
      price,
      title,
      url,
      image: image.childImageSharp.fluid,
    };
  });
  return (
    <Layout>
      <SEO title="Home" keywords={["gatsby", "application", "react"]} />
      <Box height="calc(100vh - 100px)" pt={5}>
        <Heading1 textAlign="center" mb={5}>
          Available Tours
        </Heading1>
        <Flex
          flexWrap="wrap"
          style={{ maxWidth: "1200px", margin: "0 auto" }}
        >
          {tours.map(({ title, price, description, url, image }) => (
            <TourCard
              key={title}
              title={title}
              fluidImage={image}
              price={price}
              description={description}
              url={url}
            />
          ))}
        </Flex>
      </Box>
    </Layout>
  );
};

To recap, with this method you accomplish 3 things:

You separate your data from your UI.
You avoid code duplication.
You link the images with the data.

But we still have the other requirement. To remind you, the client wants you to create pages with extra information for each tour. We conveniently ignored that requirement, but we’ll have to address it now.

In other words, you’ll have to create something like a blog where the blog post is a tour. A good solution for that requirement is to use the gatsby-mdx plugin.

Using MDX

Instead of MDX, you can use the traditional markdown files. The problem with markdown files is that all the tour “posts” will have the same structure. For example, you can’t add an interactive React component, say an accordion, in the middle of a post.

On the other hand, with MDX you can squeeze in React code if you need to. The idea here is to create an MDX file for each tour where:

The file will have frontmatter fields instead of JSON fields to store data for each tour.
For each MDX file, you will create a new page with additional information.
You will be able to add interactive elements inside the markdown text.

To use MDX in Gatsby you need to install the following packages:

yarn add @mdx-js/mdx @mdx-js/react gatsby-mdx

You can optionally install some remark packages to help with the formatting of the markdown:

yarn add gatsby-remark-images gatsby-remark-smartypants gatsby-remark-copy-linked-files gatsby-remark-external-links

You also have to configure the plugin in your gatsby-config.js:

gatsby-config.js

module.exports = {
  siteMetadata: {
    title: `Travel Agency`,
    description: `Create a travel agency site to see theme's limitations.`,
    author: `Mark`,
  },
  plugins: [
    // skipping
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: `tours`,
        path: `${__dirname}/content/tours`,
      },
    },
    {
      resolve: "gatsby-source-filesystem",
      options: {
        name: "destinations",
        path: `${__dirname}/content/destinations`,
      },
    },
    {
      resolve: `gatsby-mdx`,
      options: {
        extensions: [`.mdx`, `.md`],
        defaultLayouts: {
          default: `${__dirname}/src/components/layouts/Layout.jsx`,
        },
        gatsbyRemarkPlugins: [
          {
            resolve: "gatsby-remark-images",
            options: {
              maxWidth: 1920,
              linkImagesToOriginal: false,
            },
          },
          {
            resolve: "gatsby-remark-smartypants",
            options: {
              dashes: "oldschool",
            },
          },
          { resolve: "gatsby-remark-copy-linked-files", options: {} },
          {
            resolve: "gatsby-remark-external-links",
            options: {
              target: "_blank",
              rel: "noopener",
            },
          },
        ],
      },
    },
    // skipping
  ],
};

Now, if you place your MDX files inside the src/pages folder, the plugin will create pages for those files out of the box.

But if you do that, the URL for the tour in Spain will be https://example.com/spain-tour. I want it to be: https://example.com/tours/spain. You may also want to have unpublished tours (drafts), or the client may want to add trips to single destinations instead of only having tours. That’s why I instruct the source filesystem plugin to look inside the content/destinations folder for MDX files in the gatsby-config.js.

So, if you want to place the tour pages in a custom path, you’ll have to configure yourself the page creation in your gatsby-node.js. There are 2 steps here:

Generate slugs for the MDX pages (onCreateNode API).
Create the pages (createPages API).

I will explain what happens after the following code snippet:

gatsby-node.js

/* eslint-disable no-console */
const path = require("path");
const { createFilePath } = require("gatsby-source-filesystem");

// 1. Create those slugs from file names.
exports.onCreateNode = ({ node, actions, getNode }) => {
  const { createNodeField } = actions;

  if (node.internal.type === "Mdx") {
    const value = createFilePath({ node, getNode });

    const frontmatterTypes = ["tour", "destination"];
    const frontmatterType = node.frontmatter.type;
    if (frontmatterTypes.includes(frontmatterType)) {
      const parentPath =
        frontmatterType === "tour" ? "tours" : "destinations";
      createNodeField({
        name: "slug",
        node,
        value: `/${parentPath}${value}`,
      });
    }
  }
};

// 2. Create pages
exports.createPages = async ({ graphql, actions }) => {
  const { createPage } = actions;

  const result = await graphql(
    `
      {
        allMdx(filter: { frontmatter: { published: { eq: true } } }) {
          edges {
            node {
              id
              fields {
                slug
              }
              frontmatter {
                type
              }
            }
          }
        }
      }
    `
  );

  if (result.errors) console.log(result.errors);

  result.data.allMdx.edges.forEach(({ node }) => {
    const tourTemplate = path.resolve(`./src/templates/Tour.jsx`);
    const destinationTemplate = path.resolve(
      `./src/templates/Destination.jsx`
    );
    const template =
      node.frontmatter.type === "tour"
        ? tourTemplate
        : destinationTemplate;
    createPage({
      path: node.fields.slug,
      component: template,
      context: { id: node.id },
    });
  });
};

In the first step, you create the slugs for the MDX pages, and you store them as node fields.

In the second step, you create the pages. You get all the published MDX nodes, and, for each of them, you create a page where you pass:

The slug you created earlier as page path. This is the URL that you type in the browser to get the page.
The node id (gatsby-mdx created that) as an id field using the context. This is used by a GraphQL query in the template to get the correct page.
You specify the appropriate template depending if the MDX file is a tour or a destination.

With this configuration, you can create pages for tours and destinations. But you won’t display the last in the UI, or create MDX files for them. I added the logic just to show what you can do in a gatsby-node.js file.

The next thing you’ll have to do is to create the templates in src/templates/Tour.jsx and src/templates/Destination.jsx (The code is almost the same). I won’t list the code for the templates here, check the GitHub links.

Now, you may want to organize your MDX files in the following way. Each tour will have a folder where inside that folder will be an index.mdx file and the images the tour post needs:

content
|__ tours
|____ france
|______ france.jpg
|______ index.mdx
|____ germany
|______ germany.jpg
|______ index.mdx
|____ greece
|______ greece.jpg
|______ index.mdx
|____ japan
|______ japan.jpg
|______ index.mdx
|____ spain
|______ spain.jpg
|______ index.mdx
|____ usa
|______ usa.jpg
|______ index.mdx
|
|__ destinations

For example, the MDX file for the tour in France may look like this:

---
title: France
description: France tour
type: tour
published: true
date: "2019-06-20T15:15:10.284Z"
image: ./france.jpg
price: 800$
duration: 7 days
destination: Paris
travelers: 45+
---

## Experience France first hand

import { Box } from "../../../src/components/Primitives";

<Box p={4} bg="#1d1d1d" color="white">
  Too bored at this point to add actual content but you get the idea.
</Box>

It’s a tedious task to create all the files yourself, so I suggest to clone the repo, checkout to the mdx branch, and inspect the result.

The last two things you have to do is to change the GraphQL query and the way you map the data in the index page:

src/pages/index.jsx

// skipping imports

const IndexPage = ({ data }) => {
  const tours = data.allMdx.edges.map(({ node }) => {
    const { description, price, title, image } = node.frontmatter;
    return {
      description,
      price,
      title,
      url: node.fields.slug,
      image: image.childImageSharp.fluid,
    };
  });
  return <Layout>{/* skipping */}</Layout>;
};

export const query = graphql`
  query Tours {
    allMdx(
      filter: {
        frontmatter: { type: { eq: "tour" }, published: { eq: true } }
      }
    ) {
      edges {
        node {
          fields {
            slug
          }
          frontmatter {
            title
            price
            description
            image {
              childImageSharp {
                fluid {
                  ...GatsbyImageSharpFluid
                }
              }
            }
          }
        }
      }
    }
  }
`;

export default IndexPage;

You can also check out a similar tutorial on Egghead by Jason Lengstorf.

This was an unexpectedly long post, and I kind of regret starting it (lol). Anyway, before you leave you can see some limitations this solution has in the next section.

Limitations

The current solution with the MDX files doesn’t support multiple languages. I’m sure that a travel agency will want their content to be available in a bunch of different languages. To do that, you’ll have to change your gatsby-node.js file. I may add another section in the future (honestly, I won’t) on how to do this. At the very least, I can add a branch in the GitHub repo. For now, you can take a look at an internationalization guide in Gatsby I wrote previously.
If the client wants to edit and add new tours or destinations, the MDX files probably won’t cut it. You may have to use a CMS as your data source.

Storybook - Display code for render props components

Thu, 23 May 2019 20:05:03 GMT

Problem

You have a render props component, and you want to display the source code for it in Storybook.

For example, let’s say that you have the following Toggle Component:

Toggle.jsx

import { useState } from "react";
import PropTypes from "prop-types";

const Toggle = ({ startVisible, children }) => {
  const [visible, setVisible] = useState(startVisible);
  const toggle = () => setVisible(!visible);
  return children({ visible, toggle });
};

Toggle.propTypes = {
  startVisible: PropTypes.bool,
};

Toggle.defaultProps = {
  startVisible: false,
};

export default Toggle;

You write a story for it:

stories/Toggle.stories.jsx

import React from "react";

import { storiesOf } from "@storybook/react";

import Toggle from "../src/components/Toggle";

storiesOf("Toggle", module).add("default", () => (
  <Toggle>
    {({ visible, toggle }) => (
      <>
        <button onClick={toggle}>{visible ? "Hide" : "Show"}</button>
        {visible && (
          <p>
            Lorem ipsum dolor sit, amet consectetur adipisicing elit.
            Dicta quo expedita tempore, sunt quos quis incidunt ratione
            asperiores soluta impedit eveniet. Numquam, ipsum est maxime
            hic sit rem reprehenderit dicta!
          </p>
        )}
      </>
    )}
  </Toggle>
));

If you want to display the source code for that story, the most obvious solution is to use the Info addon. You can install it with:

yarn add -D @storybook/addon-info

and you can use it in your story:

stories/Toggle.stories.jsx

import React from "react";

import { storiesOf } from "@storybook/react";
import { withInfo } from "@storybook/addon-info"; 

import Toggle from "../src/components/Toggle";

storiesOf("Toggle", module)
  .addDecorator(withInfo) // Should be before any other decorators..
  .add("default", () => <Toggle>{/* skipping */}</Toggle>);

If you start the storybook server with yarn storybook, this is what it looks like:

Storybook addon source

As you can see, when you click the “show info” button, you can’t see the render props code.

Solution

To solve this, you can use another Storybook addon the storysource. You can install it with:

yarn add -D @storybook/addon-storysource

Import it in your .storybook/addons.js file:

.storybook/addons.js

// rest addons...
import "@storybook/addon-storysource/register";

And create a custom Webpack configuration file in .storybook/webpack.config.js:

.storybook/webpack.config.js

module.exports = function ({ config }) {
  config.module.rules.push({
    test: /\.stories\.jsx?$/,
    loaders: [require.resolve("@storybook/addon-storysource/loader")],
    enforce: "pre",
  });

  return config;
};

Now, if you restart your storybook server, you’ll see the following:

Storybook addon source

If you navigate to the “story” tab of the addons, at the bottom of the screen, you can see the correct code. Also, when you navigate between stories, the code for that story is automatically highlighted.

Implementing dark mode in React

Mon, 13 May 2019 22:12:03 GMT

In this post, you’ll see 4 methods you can use to add dark mode in a React application. You will also see their limitations and some of React’s limitations. If I were to start a new project, I would use only the last 2 methods or, even better, a library like Theme UI. Nevertheless, in my 100% unbiased opinion, I think it’s an interesting read.

Ideal solution

The ideal solution for implementing “dark mode” in a React application satisfies the following requirements:

The app supports SSR. If the app runs only on the client-side, then all solutions work without a problem. Here I use a static/prerendered app with Gatsby.
The app remembers the selected mode, so you’ll want to save it in local storage.
The app uses React’s Context API to implement dark mode. You may already be using something like styled-components themes not only for colors, but for implementing a design system, or for creating component primitives with styled-system and rebass. As a result, you may want the styling to be in one place, and you don’t want to switch between styled-components, CSS, or inline styling.

Spoiler: None of the solutions checks all the boxes above. Let’s take them one by one to see why this happens. We’ll start with the context.

This is not a step by step guide and also covers some advanced React topics. Check this GitHub repository that has the full code. I will also give a link with the relevant branch in each section.

Dark mode with context (no local storage)

GitHub repo master branch

In the first solution, you’ll use context, but you won’t save the theme in the local storage. To implement the context, I’m using a blog post by Kent C. Dodds called How to use react context effectively. (See also How to optimize your context value). The truth is that I’m not applying the performance optimizations from the linked posts because they are not relevant in our case—more about that in a bit.

Create a new file theme-context.jsx and place it in the src/context folder. The file looks like this:

src/context/theme-context.jsx

import React, {
  useState,
  useContext,
  useCallback,
  createContext,
} from "react";
import { ThemeProvider as BaseThemeProvider } from "styled-components";

import { lightTheme, darkTheme } from "../themes";

const ThemeContext = createContext();

const ThemeProvider = ({ children }) => {
  const [themeString, setThemeString] = useState("light");
  const themeObject = themeString === "dark" ? darkTheme : lightTheme;
  return (
    <ThemeContext.Provider value={{ themeString, setThemeString }}>
      <BaseThemeProvider theme={themeObject}>
        {children}
      </BaseThemeProvider>
    </ThemeContext.Provider>
  );
};

function useTheme() {
  const context = useContext(ThemeContext);
  if (!context)
    throw new Error("useTheme must be used within a ThemeProvider");
  const { themeString, setThemeString } = context;
  const toggleTheme = useCallback(() => {
    if (themeString === "light") setThemeString("dark");
    else if (themeString === "dark") setThemeString("light");
  }, [themeString]);
  return {
    theme: themeString,
    toggleTheme,
  };
}

export { ThemeProvider, useTheme };

Let’s try to break down what’s going on in this file. In short, we’re creating a new context for the theme that we can use inside our components with a hook. More specifically:

We create a new component that stores the current theme in the state. The current theme is a string that can be "light" or "dark". We call the component ThemeProvider but think of it as a regular React component that has some state and renders some children.
We use the theme from the state as the value for a new context (ThemeContext). We wrap the children of the previous component with the ThemeContext.Provider (See how to use context in React docs).
In the same component, we use the theme provider from styled-components (BaseThemeProvider). We wrap the children of the component with the BaseThemeProvider and synchronize the two themes when the first changes. We synchronize the two themes with the themeObject variable which is also the value for the styled-components context.
We’ll wrap Gatsby’s root element with the ThemeProvider component (in a bit) in gatsby-browser.js and gatsby-ssr.js files.
Finally, we create a hook that uses the new context. Inside that hook, we show an error if someone tries to use the context outside of a provider (see the previous step). We also memoize the toggleTheme function with a useCallback hook, but, in this case, it’s not that important because we won’t pass that function as a prop to a component. You can remove it if you want. To use the theme context in a component, you only have to use the hook.

You might be wondering why are we creating a new context for the current theme, instead of using the context from styled-components. We do that because we want inside the context value the function that toggles the theme, something that the context from styled-components doesn’t seem to offer.

Let’s now talk about the performance optimizations we skipped—you can safely skip this paragraph. The value we pass to the theme context is an object. We pass it with value={{ themeString, setThemeString }}. This means that we are creating and passing a different object in each render. The components that use the context render when the value of the context changes. In a hypothetical scenario where a parent of the ThemeProvider component renders, the ThemeProvider component should also render. Because the value of the context is different in every render, that means that the components that use the context should also render. These would be some unnecessary renders because the theme didn’t change. For this reason, many people memoize the context value with the useMemo hook to ensure that it doesn’t change when the theme is the same. Additionally, they wrap the components that use the theme with memo that prevents re-renders when the props are the same. See this codesandbox from Kent C. Dodds that illustrates that. In our case, we’re not wrapping the ThemeProvider component in a parent component that renders all the time. The only time the ThemeProvider renders is when the theme changes, and, in this case, we want the components that use the context to render. That’s why I skipped the performance optimizations.

Next, you want to create the styled-components theme objects in src/themes/ folder. I said that I want to use the theme for other things, not only for colors but let’s use them only for colors to keep the code simple:

src/themes/index.js

const lightTheme = {
  bg: "white",
  bgDark: "pink",
  color: "black",
  accent: "blue",
};

const darkTheme = {
  bg: "#2c1320",
  bgDark: "#15090f",
  color: "white",
  accent: "#ef86a9",
};

export { lightTheme, darkTheme };

Then, you wrap the Gatsby’s root element in gatsby-browser.js and gatsby-ssr.js. The files are identical; just copy-and-paste the same code:

gatsby-ssr.js/gatsby-browser.js

import React from "react";
import "normalize.css";
import "typeface-fira-sans";
import "typeface-merriweather";

import { ThemeProvider } from "./src/context/theme-context";
import GlobalStyle from "./src/components/GlobalStyle";

export const wrapRootElement = ({ element }) => (
  <ThemeProvider>
    <>
      <GlobalStyle />
      {element}
    </>
  </ThemeProvider>
);

You also want a Toggle component that will switch between light and dark mode. For this reason, I copied the Toggle component from Dan Abramov’s blog, which is a slightly altered Toggle from react-toggle. In the comments, it says that it has some accessibility improvements. So you can either copy the Toggle.js, Toggle.css, and add the sun/moon images from Dan Abramov’s blog, or just use the default toggle from react-toggle.

You’ll place the toggle somewhere in the Header component. I’m also using the theme colors, in the Container component, and the theme hook:

src/components/header.js

import React from "react";
import { Link } from "gatsby";
import styled from "styled-components";
import PropTypes from "prop-types";
import Toggle from "./Toggle";

import sun from "../images/sun.png";
import moon from "../images/moon.png";
import { useTheme } from "../context/theme-context"; 

const Container = styled.header`
  color: ${({ theme }) => theme.color};
  background-color: ${({ theme }) => theme.bgDark};
  margin-bottom: 1.45rem;

  a {
    text-decoration: none;
    color: ${({ theme }) => theme.color};
  }
`;

const Header = ({ siteTitle }) => {
  const { theme, toggleTheme } = useTheme(); 
  return (
    <Container>
      <div
        style={{
          margin: `0 auto`,
          maxWidth: 960,
          padding: `1.45rem 1.0875rem`,
        }}
      >
        <h1 style={{ margin: 0 }}>
          <Link to="/">{siteTitle}</Link>
        </h1>
        <Toggle
          defaultChecked={theme === "dark" ? true : false}
          onChange={toggleTheme}
          icons={{
            checked: (
              <img
                style={{ pointerEvents: "none" }}
                width="16"
                height="16"
                alt="moon"
                aria-hidden
                src={moon}
              />
            ),
            unchecked: (
              <img
                style={{ pointerEvents: "none" }}
                width="16"
                height="16"
                alt="sun"
                aria-hidden
                src={sun}
              />
            ),
          }}
        />
      </div>
    </Container>
  );
};

Header.propTypes = {
  siteTitle: PropTypes.string,
};

Header.defaultProps = {
  siteTitle: ``,
};

export default Header;

And this is what the app looks like after the changes:

Dark mode with context

It works fine in both development and production modes, but every time you reload the page, it defaults back to light mode. If your app doesn’t reload much, it might be ok; but you never know how the user will use your app. As a result, this behavior can become annoying really fast. To fix that, you can save the preferred user theme in local storage:

Dark mode with context and local storage

GitHub repo context-local-storage-bug branch

I will use a custom hook to save the theme string in local storage. Alternatively, you can use an NPM package for that. You can use the useLocalStorage hook by giving a name string (theme) and the initialValue (“light”). It returns the value and a function to change it. The hook keeps in the state the value, and when the value changes, it runs a side-effect to save the value in the local storage. It initializes the state with the existing value in local storage or with the initialValue from the user. It also has some guards for SSR (windowGlobal variable).

src/hooks/useLocalStorage.js

import { useState, useEffect } from "react";

export const useLocalStorage = (name, initialValue) => {
  const windowGlobal = typeof window !== "undefined" && window;
  const [value, setValue] = useState(() => {
    if (windowGlobal) {
      const currentValue = windowGlobal.localStorage.getItem(name);
      return currentValue ? JSON.parse(currentValue) : initialValue;
    }
    return initialValue;
  });

  useEffect(() => {
    if (windowGlobal)
      windowGlobal.localStorage.setItem(name, JSON.stringify(value));
  }, [name, value, windowGlobal]);
  return [value, setValue];
};

To use this hook in the ThemeProvider, you only have to change 2 lines of code:

src/context/theme-context.jsx

import React, { useContext, useCallback, createContext } from "react";
import { ThemeProvider as BaseThemeProvider } from "styled-components";

import { useLocalStorage } from "../hooks/useLocalStorage"; 
import { lightTheme, darkTheme } from "../themes";

const ThemeContext = createContext();

const ThemeProvider = ({ children }) => {
  const [themeString, setThemeString] = useLocalStorage(
    "theme",
    "light"
  );
  const themeObject = themeString === "dark" ? darkTheme : lightTheme;
  return (
    <ThemeContext.Provider value={{ themeString, setThemeString }}>
      <BaseThemeProvider theme={themeObject}>
        {children}
      </BaseThemeProvider>
    </ThemeContext.Provider>
  );
};

function useTheme() {
  // skipping hook implementation
}

export { ThemeProvider, useTheme };

If you try it in development, it works fine. You can switch the theme by pressing the toggle, and if you reload the page, the app remembers your preference. But if you build it with gatsby build, and serve it locally with gatsby serve, you’ll see the following:

The bug with context and local storage

If you choose the dark mode, the theme is getting stored correctly in local storage, but if you reload the page you see the light mode. Also, the Toggle component is now broken. It needs some extra clicks to change the theme after reloading the page (if you had dark mode selected). If you log in the console the themes and context, you’ll see that everything is fine; the dark mode is selected as expected. So, why is this happening?

This happens because of how React’s hydrate method treats differences between client and server. During build time, we apply the default light theme, so we add to the DOM elements the styled-components classes for the light theme. When we go to the client and try to hydrate the markup, React sees that we now have a different theme from local storage, so we want to apply different classes for the dark mode. React doesn’t do that for attributes. Only for text mainly because of timestamps differences.

You can see that if you create a post context with 3 posts, and store them the same way you store the theme in local storage. After you save them in local storage, you can comment one post out to create a difference between the server (3 posts) and the client (2 posts). You’ll see that React will hydrate the text correctly, as it’s shown in the following video:

GitHub repo posts branch

React hydrates the text correctly

In this case, you can force an extra render when the component mounts to update the classes. If you do that, this is what the app looks like in production:

Forcing an extra render

GitHub repo context-local-storage (correct)

It works now, but you momentarily see a flash of unstyled content. This happens because you go quickly from light to dark mode. It’s quite noticeable because the color differences are big. You can tone it down if you add a CSS transition in the items that change color, but you’ll have to do this for every item. Also, if your app takes some time to load, or the users access your app from a slow network, they will be stuck for a while with the incorrect theme until the app becomes interactive.

It seems that you can’t have a perfect solution if you only use context. In the following method, you’ll run some code before React to manipulate the DOM and change the colors with CSS variables:

Dan Abramov’s solution with CSS

In this section, I will only give a brief overview of this method. If you want the full code, please check the CSS branch on the GitHub repository

You saw earlier that you can’t rely on context due to the way React’s hydrate method treats differences between client and server. So you’ll have to do something else. You’ll want to run your JavaScript code that will initialize the dark mode in a script tag (before React loads). You’ll place that script right after the opening body tag because you want the body element to exist, but you don’t want the browser to render something on the screen and see that flash again. This works because inline scripts block rendering. In a Gatsby application, one of the ways you can add a script tag is by overriding the default html.js file.

Inside that script, you declare an IIFE where you do the following:

You get the saved theme from local storage and save the correct one if necessary.
You perform a matchMedia query for a preferred color scheme. If the user has a preferred color scheme, you aggregate the saved and the preferred, giving priority to the saved.
You add a class to the body element for the preferred mode (dark, light). You declare your colors with CSS custom properties under body.light and body.dark (meaning that the colors will change when the class of the body changes).
You attach to the window object the current theme and some methods that you’ll use inside your header component later to change the theme.

src/html.js

import React from "react";
import PropTypes from "prop-types";

export default class HTML extends React.Component {
  render() {
    return (
      <html {...this.props.htmlAttributes}>
        <head>
          <meta charSet="utf-8" />
          <meta httpEquiv="x-ua-compatible" content="ie=edge" />
          <meta
            name="viewport"
            content="width=device-width, initial-scale=1, shrink-to-fit=no"
          />
          {this.props.headComponents}
        </head>
        <body {...this.props.bodyAttributes} className="light">
          <script
            dangerouslySetInnerHTML={{
              __html: `
              (function() {
                window.__onThemeChange = function() {};
                function setTheme(newTheme) {
                  window.__theme = newTheme;
                  preferredTheme = newTheme;
                  document.body.className = newTheme;
                  window.__onThemeChange(newTheme);
                }
                var preferredTheme;
                try {
                  preferredTheme = localStorage.getItem('theme');
                } catch (err) { }
                window.__setPreferredTheme = function(newTheme) {
                  setTheme(newTheme);
                  try {
                    localStorage.setItem('theme', newTheme);
                  } catch (err) {}
                }
                var darkQuery = window.matchMedia('(prefers-color-scheme: dark)');
                darkQuery.addListener(function(e) {
                  window.__setPreferredTheme(e.matches ? 'dark' : 'light')
                });
                setTheme(preferredTheme || (darkQuery.matches ? 'dark' : 'light'));
              })();
            `,
            }}
          />
          {this.props.preBodyComponents}
          <div
            key={`body`}
            id="___gatsby"
            dangerouslySetInnerHTML={{ __html: this.props.body }}
          />
          {this.props.postBodyComponents}
        </body>
      </html>
    );
  }
}

HTML.propTypes = {
  htmlAttributes: PropTypes.object,
  headComponents: PropTypes.array,
  bodyAttributes: PropTypes.object,
  preBodyComponents: PropTypes.array,
  body: PropTypes.string,
  postBodyComponents: PropTypes.array,
};

This is the code outside of the template string with comments that explain what it does:

(function () {
  // This is a callback that gets called by setTheme.
  // You’ll assign a setState function to this callback to
  // update the React state later inside the header component.
  window.__onThemeChange = function () {};
  // An inner function that changes the mode but
  // doesn’t save in local storage. You won’t use this.
  function setTheme(newTheme) {
    // This is used inside the header component
    // to set the initial state. It’s also used to keep track
    // of the mode when the header component mounts/unmounts.
    window.__theme = newTheme;
    preferredTheme = newTheme;
    document.body.className = newTheme;
    window.__onThemeChange(newTheme);
  }
  // Get the saved theme from local storage
  var preferredTheme;
  try {
    preferredTheme = localStorage.getItem("theme");
  } catch (err) {}
  // The function that changes the mode and saves
  // to local storage. You’ll use that inside the header
  // component.
  window.__setPreferredTheme = function (newTheme) {
    setTheme(newTheme);
    try {
      localStorage.setItem("theme", newTheme);
    } catch (err) {}
  };
  var darkQuery = window.matchMedia("(prefers-color-scheme: dark)");
  // When the user changes the preferred color scheme,
  // call the window.__setPreferredTheme method.
  darkQuery.addListener(function (e) {
    window.__setPreferredTheme(e.matches ? "dark" : "light");
  });
  // Set the theme for the first time by aggregating the
  // saved theme (preferred theme variable) and the color
  // scheme query. Don't forget the parenthesis, they are
  // important
  setTheme(preferredTheme || (darkQuery.matches ? "dark" : "light"));
})();

Next, go to the Header component (because that’s where the Toggle component is) and do the following:

Get the correct theme from window.__theme in an effect that runs only on mount.
When the user clicks the Toggle to change the theme, you call the window.__setPreferredTheme() method. This method saves the theme in localStorage, and updates the body class and the React state.
The window.__onThemeChange() is a callback that gets called by window.setPreferredTheme() and updates the React state in this case.

src/components/header.jsx

import React, { useState, useEffect } from "react";
// skipped

const Container = styled.header`
  color: var(--color);
  background-color: var(--bgDark);
  margin-bottom: 1.45rem;

  a {
    text-decoration: none;
    color: var(--color);
  }
`;

const Header = ({ siteTitle }) => {
  const [theme, setTheme] = useState(null);
  useEffect(() => {
    setTheme(window.__theme);
    window.__onThemeChange = () => setTheme(window.__theme);
  }, []);
  return (
    <Container>
      <div
        style={{
          margin: `0 auto`,
          maxWidth: 960,
          padding: `1.45rem 1.0875rem`,
        }}
      >
        <h1 style={{ margin: 0 }}>
          <Link to="/">{siteTitle}</Link>
        </h1>
        {theme ? (
          <Toggle
            checked={theme === "dark"}
            onChange={(e) =>
              window.__setPreferredTheme(
                e.target.checked ? "dark" : "light"
              )
            }
            
          />
        ) : (
          <div style={{ height: "28px" }} />
        )}
      </div>
    </Container>
  );
};

// skipped

Because you don’t want to render the Toggle with the wrong state on mount, you render a placeholder div element if the theme is null. The theme is null on mount both in development and production, which is a good thing because you don’t want differences between the development and production. An alternative is to use the useLayoutEffect instead of useEffect that will run the effect before React renders the component with the wrong state on screen. This is not a good practice, though, because it delays browser paints. So pretend I didn’t mention it and stick with the first option.

Now, you can use CSS variables for the colors instead of getting them from the styled-components theme. I define those variables in a GlobalStyle.jsx component in this example, but you can import a regular CSS file if you want:

src/components/GlobalStyle.jsx

import { createGlobalStyle } from "styled-components";

const GlobalStyle = createGlobalStyle`
    // skipped

    a {
        color: var(--accent);
    }

    body {
        background-color: var(--bg);
    }

    body.light {
      --bg: white;
    --bgDark: pink;
    --color: black;
    --accent: blue;
    }

    body.dark {
      -webkit-font-smoothing: antialiased;

    --bg: #2c1320;
    --bgDark: #15090f;
    --color: white;
    --accent: #ef86a9;
    }
`;
export default GlobalStyle;

If you build the application, this is how it looks. P.S. Not exactly, in the following video I render the Toggle in the wrong state on mount. You can see that when I change the theme to dark mode and refresh the page. Don’t worry though, the code above doesn’t do that.

The solution with CSS variables

By the way, in this example, I’m still wrapping the root element with a ThemeProvider from styled-components. I just don’t use the theme for the colors. See an out-of-the-box dark mode solution with Theme UI if you don’t like that. I should also note that CSS variables (or CSS custom properties) do not work on Internet Explorer.

A refactor of the CSS solution

GitHub repo css-refactor branch

When I first wrote this article more than a year ago, I treated Dan’s solution as a black box without explaining it much which is something I didn’t like. I made some edits in the previous section, but I also want to show you a refactor of this method.

This section needs a rewrite (a rewrite of the rewrite) because after writing it, I experimented a bit more and pushed to the repo more branches that use a context provider that I think are better. Check the GitHub repo (master branch) for those solutions.

The gist of this method is that you want to run the code that switches the mode in a script. In other words, you take the control from React and give it to an inline script. You do that because React doesn’t handle this use case well. In the next snippet, you can see the script code with syntax highlighting, outside of a string prop:

(function () {
  window.__onThemeChange = function () {};
  function setTheme(newTheme) {
    window.__theme = newTheme;
    preferredTheme = newTheme;
    document.body.className = newTheme;
    window.__onThemeChange(newTheme);
  }
  var preferredTheme;
  try {
    preferredTheme = localStorage.getItem("theme");
  } catch (err) {}
  window.__setPreferredTheme = function (newTheme) {
    setTheme(newTheme);
    try {
      localStorage.setItem("theme", newTheme);
    } catch (err) {}
  };
  var darkQuery = window.matchMedia("(prefers-color-scheme: dark)");
  darkQuery.addListener(function (e) {
    window.__setPreferredTheme(e.matches ? "dark" : "light");
  });
  setTheme(preferredTheme || (darkQuery.matches ? "dark" : "light"));
})();

I find it a bit complicated, so, in the next snippet, you can find a more straight-forward (?) version with comments that explain what happens in each step:

// I use regular functions and var instead of
// const to support older browsers (not IE).
// I also give names to the functions for
// easier debugging.
(function initializeTheme() {
  // 1. Get the existing theme from local storage
  // and save it to window.__theme.
  try {
    window.__theme = localStorage.getItem("theme");
  } catch (err) {
    console.log("Couldn’t get the theme from local storage.");
  }

  // 2. Inside React, you “subscribe” to theme updates by
  // overwriting the following callback.
  window.__onThemeChangeCallback = function () {};

  // 3. The function that changes the theme and
  // calls the callback.
  window.__setTheme = function setTheme(newTheme) {
    // You’ll use that variable to set the initial
    // React state. It's also used to store the
    // correct mode when the component mounts
    // and unmounts.
    window.__theme = newTheme;
    // Change the class, call the callback, and
    // save the theme in local storage.
    document.body.className = newTheme;
    window.__onThemeChangeCallback(newTheme);
    try {
      localStorage.setItem("theme", newTheme);
    } catch (err) {
      console.log("Couldn’t save the theme in local storage.");
    }
  };

  // 4. A color scheme listener that calls the setTheme
  // method.
  var darkQuery = window.matchMedia("(prefers-color-scheme: dark)");
  darkQuery.addListener(function darkQueryListener(e) {
    window.__setTheme(e.matches ? "dark" : "light");
  });

  // 5. Set the theme for the first time according to
  // the saved theme and the color scheme query.
  window.__setTheme(
    window.__theme || (darkQuery.matches ? "dark" : "light")
  );
})();

A problem with both of those snippets is that you can’t use the code inside the Toggle for more than one component. You can’t extract it to a useDarkMode hook for example because every time you assign a callback to onThemeChangeCallback, inside a React component, you overwrite the previous component’s callback. As a result, their state will get out of sync. If you want to use the theme only inside a Toggle component, you don’t have a problem. You can try to solve this by using the theme in a single context component, but you may face the same problems with hydration all over again. So instead of overwriting a single callback, you can keep track of a callback array, and then iterate through it inside the script and call them one-by-one. You can also overengineer the problem with an observer pattern.

(function initializeTheme() {
  try {
    window.__theme = localStorage.getItem("theme");
  } catch (err) {
    console.log("Couldn’t get the theme from local storage.");
  }

  var callbacks = [];
  window.__addCallback = function (cb) {
    callbacks.push(cb);
  };
  window.__removeCallback = function (cb) {
    callbacks = callbacks.filter(function (callback) {
      return callback !== cb;
    });
  };

  window.__setTheme = function setTheme(newTheme) {
    window.__theme = newTheme;
    document.body.className = newTheme;
    callbacks.forEach(function (cb) {
      cb();
    });
    try {
      localStorage.setItem("theme", newTheme);
    } catch (err) {
      console.log("Couldn’t save the theme in local storage.");
    }
  };

  var darkQuery = window.matchMedia("(prefers-color-scheme: dark)");
  darkQuery.addListener(function darkQueryListener(e) {
    window.__setTheme(e.matches ? "dark" : "light");
  });

  window.__setTheme(
    window.__theme || (darkQuery.matches ? "dark" : "light")
  );
})();

This how to use the code above inside the Header component.

src/components/header.js

useEffect(() => {
  setTheme(window.__theme);
  const callback = () => setTheme(window.__theme);
  window.__addCallback(callback);
  return () => {
    window.__removeCallback(callback);
  };
}, []);

src/components/header.js

<Toggle
  checked={theme === "dark"}
  onChange={(e) =>
    window.__setTheme(e.target.checked ? "dark" : "light")
  }
/>

I use the last solution in this blog, and, as far as I know, it works. The only difference is that I don’t use a Layout component that unmounts on page change. Instead, I wrap Gatsby’s pageElement with the Layout.

Displaying hundreds of images with react-window and gatsby-image

Sat, 30 Mar 2019 22:12:03 GMT

This post is a step-by-step guide on how to use gatsby-image and react-window to display a large number of images. The app we’re going to build displays in a grid all the available Hearthstone cards. We choose Hearthstone cards because well… I like Hearthstone and because at this moment, there are over 1800 cards. That translates to a lot of images we’ll have to display, so it’ll be a good exercise. The completed project is available on GitHub if you want to take a look. We’ll also see some limitations our approach has along the way.

Get the project

If you want to follow along, clone the project, and checkout to the start branch. In this branch, we’ve used the default Gatsby starter, removed some unused files, edited some styles, and finally, we added the data and the images. It has over 1800 images (about 90mb), so it may take a while to download.

# Get the project
git clone https://github.com/MarkosKon/gatsby-image-react-window
cd gatsby-image-react-window
git checkout start

Install and configure

The next thing we want to do is to install our dependencies. The default Gatsby starter has already installed and configured all the image plugins we need. We only need to install gatsby-transformer-json which will parse our data. We’ll also install 3 libraries for react-window and lodash.sortby to sort our data:

# Install dependencies
yarn add gatsby-transformer-json
yarn add react-window react-virtualized-auto-sizer memoize-one
yarn add lodash.sortby

After we install our dependencies, we’ll configure the transformer JSON plugin. This is a transformer plugin that transforms the output of the gatsby-source-filesystem into something more useful. As a result, we’ll have to point a folder to the source plugin (src/data/cards), and give our data a name (cards) that we’ll use later in our GraphQL queries. To do that, we open the gatsby-config.js file, and we add the highlighted lines:

gatsby-config.js

module.exports = {
  siteMetadata: { /* ...skipping stuff */}
  plugins: [
    // ...skipping stuff
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: `cards`,
        path: `${__dirname}/src/data/cards`
      }
    },
    `gatsby-transformer-sharp`,
    `gatsby-plugin-sharp`,
    `gatsby-transformer-json`
    // ...skipping stuff
  ]
};

The transformer JSON plugin will prepare for us 2 GraphQL types: allCardsJson and cardsJson. We’ll use the first. You can play around with the GraphQL queries in the GraphiQL server if you navigate to http://localhost:8000/___graphql. Remember that we have thousands of images, so it may take a while to process them as we’ll see soon (spoiler: it took me 7 minutes).

Magic tricks

I want to mention here a cool trick we do with our data which I blatantly copied from gatsbygram. If you take a look at src/data/cards, you’ll see that we have a cards.json file with our card data and an images folder with our image files. If you open cards.json, you’ll see an image field that points to the images folder e.g.

{
  "name": "Ice Lance",
  "cost": 1,
  "dbfId": 172,
  "image": "images/188.png"
}

With this sorcery, we can bind our imageSharp fields (we’ll see them shortly) directly into our data. I don’t know which plugin is responsible for that (I suspect gatsby-transformer-json), but I wish I knew that sooner.

We can now start the development server to see if everything works with gatsby develop. If you already opened the server, you’ll have to restart it.

The GraphQL query

In our index page, we’ll write our GraphQL query that will load all the data and the images. We’ll pass that data into a Cards component. That component will have a FixedSizeGrid from react-window and a cell that will display the actual card images.

But before we continue, let’s talk about a compromise we’ll have to make. It has to do with our bundle size and the progressive loading of the images.

If we use the default GraphQL fragment from the available gatsby-image fragments (a fragment is a collection of fields that’s used for convenience, alternatively we could require each field manually), say GatsbyImageSharpFluid, our bundle size will end up at 3.5mb! This happens because Gatsby adds the query data into a separate JSON file that gets downloaded after we load the page. In that file, we’ll also get, as an object field, the base64 inline image that’s used for the Medium-style progressive loading. For 3-4 images that extra size is ok. But for hundreds or even thousands of images, we’ll just have to forget the progressive loading. The same thing is true for the traced SVG technique (and worse) if you’re wondering. So we’ll fall back to the simple fade-in, and use the GatsbyImageSharpFluid_noBase64 fragment.

Also, because we have about 1800 images, sharp will take a while to process our images. For example, it took me about 7 minutes to generate the 3700+ required thumbnails.

Now, let’s get our data, and log them into the console. Open the index.js file inside src/pages, and add the highlighted lines:

src/pages/index.js

import React from "react";
import { graphql } from "gatsby";
import sortBy from "lodash.sortby";

import Layout from "../components/layout";
import SEO from "../components/seo";
// import Cards from "../components/Cards"

const IndexPage = ({ data }) => {
  const cards = data.allCardsJson.edges.map((n) => n.node);
  const sortedCards = sortBy(cards, ["cost", "name"]);
  console.log(sortedCards);
  return (
    <Layout>
      <SEO title="Home" keywords={[`gatsby`, `application`, `react`]} />
      <h1>Hi people</h1>
      <p>Welcome to your new Gatsby site.</p>
      <p>Now go build something great.</p>
      {/* <Cards cards={sortedCards} /> */}
    </Layout>
  );
};

export const query = graphql`
  query indexPage {
    allCardsJson {
      edges {
        node {
          dbfId
          name
          cost
          image {
            childImageSharp {
              fluid {
                ...GatsbyImageSharpFluid_noBase64
              }
            }
          }
        }
      }
    }
  }
`;

export default IndexPage;

If we want to see the JSON file we mentioned earlier, we’ll have to build the app with gatsby build and open the Network tab of the Chrome DevTools. But I’ve already done this for you. In the next image, you can see the size of the file which is 102kb for our query (which is acceptable for so many images):

The file with the image data

And in the following image, you can see what the object looks like if you open it in the preview:

What's inside that file

Add FixedSizedGrid from react-window

We have our data; let’s now create the actual grid. As we said earlier, we’ll use the FixedSizeGrid from react-window. We’ll also use the standalone AutoSizer component from react-virtualized. This component will make our grid responsive and will try to fill all the available space, so we don’t have to specify a static grid width and height.

We start with the Grid, and later we’ll add the Cell. Create a new file src/components/Cards.jsx, paste the following code, and we’ll talk about it later:

src/components/Cards.jsx

import React from "react";
import { FixedSizeGrid as Grid } from "react-window";
import AutoSizer from "react-virtualized-auto-sizer";

import "./cards.css";

const Cards = ({ cards }) => (
  <div
    style={{
      minHeight: "100vh",
      backgroundColor: "#d6cae2",
      marginTop: "2em",
      position: "sticky",
      top: "0px",
    }}
  >
    <AutoSizer defaultWidth={1920} defaultHeight={1080}>
      {({ width, height }) => {
        const cardWidth = 286;
        const cardHeight = 395;
        const columnCount = Math.floor(width / cardWidth);
        const rowCount = Math.ceil(cards.length / columnCount);
        return (
          <Grid
            className="grid"
            width={width}
            height={height}
            columnCount={columnCount}
            columnWidth={cardWidth}
            rowCount={rowCount}
            rowHeight={cardHeight}
            itemData={{ cards, columnCount }}
          >
            {Cell}
          </Grid>
        );
      }}
    </AutoSizer>
  </div>
);

export default Cards;

Also, create a CSS file src/components/cards.css that we’ll use to center the cards inside the Grid:

src/components/cards.css

/* to center cards inside grid */
.grid > div {
  position: relative;
  margin: auto;
}

Let’s explain what we do in the code above:

In the AutoSizer component, we pass a defaultWidth and a defaultHeight. That’s useful at build time when we don’t have the window object available.
We’re calculating the number of columns and rows based on the number of cards we have and their size in pixels (columnCount, rowCount).
The “grid” className prop is used for centering the cards with CSS. We’re specifying the columnWidth and rowHeight to be equal to the cardWidth and cardHeight respectively. Finally, we pass with the itemData prop the cards and the columnCount down to the Cell component that we’ll see shortly.

For this example, we don’t have to create a separate file for the Cell. It would have made more sense to do that if we were developing a more complex app. So in the same file, we continue with the Cell and the gatsby-image components:

src/components/Cards.jsx

import React from "react";
import { FixedSizeGrid as Grid } from "react-window";
import AutoSizer from "react-virtualized-auto-sizer";
import GatsbyImage from "gatsby-image"

import "./cards.css";

const Cell = ({ columnIndex, rowIndex, style, data }) => {
  const { cards, columnCount } = data
  const singleColumnIndex = columnIndex + rowIndex * columnCount
  const card = cards[singleColumnIndex]

  return (
    <div style={style}>
      {card && (
        <div
          key={card.dbfId}
          style={{
            width: "286px",
            height: "395px",
            display: "inline-block",
          }}
        >
         <GatsbyImage
            title={card.name}
            fluid={card.image.childImageSharp.fluid}
          />
        </div>
      )}
    </div>
  )
}

const Cards = ({ cards }) => (/* ...skipping Cards */);

export default Cards;

Let’s explain what happens here:

The Cell component, which represents each card, takes from the Grid 3 standard props: columnIndex, rowIndex, and style. It also takes the optional data prop which comes from the itemData we mentioned earlier.
We see that we apply that style prop to the container. The style prop positions our cells inside the grid.
Because the Grid gives us only the columnIndex and the rowIndex, but our cards are in a one-dimensional array, we’re doing a conversion with the singleColumnIndex variable.
Sometimes, the cards may not fill the last row. In this case, the card would be undefined because the singleColumnIndex will point out of the array. Additionally, we’ll render an empty container, and not bother rendering the card stuff (I’m talking about the card && check).

Now, if we want to see the result, we’ll have to comment out the 2 “cards” lines we added earlier in the index page, and remove the console.log statement. In the following image, we can see the result:

The card grid

One last thing we want to do is to optimize the performance of our Grid.

Optimize performance

We already have a performance optimization in our Grid. We avoid passing an anonymous function as a cell. When we do that, the cell mounts/unmounts between renders, and that’s a problem when we render complex cells. That’s why the itemData prop exists by the way. It allows us to pass data down to our cells instead of doing that with an anonymous component and composition.

The first thing we want to do is to make our cell a memo component (aka pure component). On top of that, we’ll compare its props with the areEqual function (the equivalent of shouldComponentUpdate for functional components) from react-window to skip renders if the complex props like the style don’t change (deep equality check instead of shallow from the pure component).

You can add the highlighted lines in the src/components/Cards.jsx file:

src/components/Cards.jsx

import React, { memo } from "react";
import GatsbyImage from "gatsby-image";
import { FixedSizeGrid as Grid, areEqual } from "react-window";
import AutoSizer from "react-virtualized-auto-sizer";

import "./cards.css";

const Cell = memo(({ columnIndex, rowIndex, style, data }) => {
  const { cards, columnCount } = data
  const singleColumnIndex = columnIndex + rowIndex * columnCount
  const card = cards[singleColumnIndex]

  return (
      /* ...skipping render */
      )
}, areEqual);

// ...skipping the rest

You can optionally add memoization for the itemData, although I believe this is not necessary for our application. Anyway, this is how you do with the memoize-one library:

src/components/Cards.jsx

import React, { memo } from "react";
import GatsbyImage from "gatsby-image";
import { FixedSizeGrid as Grid, areEqual } from "react-window";
import AutoSizer from "react-virtualized-auto-sizer";
import memoize from "memoize-one";

import "./cards.css";

const createItemData = memoize(({ columnCount, cards }) => ({
  columnCount,
  cards,
}));

const Cell = memo(({ columnIndex, rowIndex, style, data }) => {
  /* ...skipping Cell */
}, areEqual);

const Cards = ({ cards }) => (
  <div
    style={{
      minHeight: "100vh",
      backgroundColor: "#d6cae2",
      marginTop: "2em",
      position: "sticky",
      top: "0px",
    }}
  >
    <AutoSizer defaultWidth={1920} defaultHeight={1080}>
      {({ width, height }) => {
        const cardWidth = 286;
        const cardHeight = 395;
        const columnCount = Math.floor(width / cardWidth);
        const rowCount = Math.ceil(cards.length / columnCount);
        const itemData = createItemData({ cards, columnCount });
        return (
          <Grid
            className="grid"
            width={width}
            height={height}
            columnCount={columnCount}
            columnWidth={cardWidth}
            rowCount={rowCount}
            rowHeight={cardHeight}
            itemData={itemData}
          >
            {Cell}
          </Grid>
        );
      }}
    </AutoSizer>
  </div>
);

export default Cards;

For more performance tips, check this YouTube video from the creator of the react-window library. In the next section, I compiled a list with some limitations and potential problems.

Caveats

We have to forget about the progressive loading feature of the gatsby-image component due to bundle size. If we could configure gatsby-plugin-sharp to put the placeholder images in a separate file instead of adding them as inline base64 strings in a JSON file, this would not be a problem.
If a user scrolls down the grid really fast, they will trigger a ton of downloads, and that will compromise the user experience and our bandwidth. I’m talking about server bandwidth, but the same is true for user data. You can see that in action if you add a Network throttle in the Performance tab of Chrome DevTools.
If, for some reason, we can’t combine our data with the images, as we showed earlier, and they are in separate objects, we’ll have to search for the image on runtime, with a small performance overhead:

// Getting the data from the GraphQL queries (index page).
const cards = data.allCardsJson.edges.map((n) => n.node);
const images = data.allImageSharp.edges.map((n) => n.node);

// Finding the image (cell component).
const image = images.find((img) =>
  img.node.fluid.src.includes(`/${cardToRender.imageUrl}`)
).node.fluid;

A performance problem when you combine useReducer and useContext

Wed, 27 Mar 2019 22:12:03 GMT

TL;DR: Instead of choosing redux and react-redux for state management, I created an AppContainer component that internally uses a useReducer hook, and shares the dispatch/state with the useContext hook. This can lead to performance problems for components that use the context, but when the context changes, they don’t need to render again.

To solve this, I broke the component into 2 parts; into a parent component that uses the context, and a child component that takes the context data as props. This way, we can use PureComponent (memo) and shouldComponentUpdate (areEqual) on the expensive child to skip unnecessary renders. It’s the second solution our prophet gives in this GitHub issue. Alternatively, you can use redux and react-redux…

We’ll also see the steps I took to understand the problem, measure, and finally optimize the code. As you can imagine, I didn’t come up with all this, but instead, I read the work of other brilliant people.

Disclaimers

I want to say here that I don’t have a clue if the React team approves this pattern (useReducer + useContext instead of React + react-redux). The hooks are still new, and the documentation not complete. So, it’s expected that we, as developers, we’ll find a way to misuse them and embarrass ourselves in the process.

Anyway… I used it because I wanted a side project to experiment with React hooks and more specifically with the useReducer hook. I searched in Google, and some guides (when I say guides I mean blog posts) suggested using it alongside the useContext hook. I liked that because I would not have to “mapStateToProps” or “mapDispatchToProps”, which is something that annoys me, but instead, I would just require stuff from the useContext hook. With that out of the way, let’s see what this app does.

A quick description of the app

Hearthstone is an online card game created by Blizzard. HSFilter, on the other hand, gives you the ability to filter Hearthstone cards based on attributes you choose, and then add them to a deck (max 30 cards) that you can use to play against your opponents. It’s built with Gatsby if you’re wondering. Let’s now proceed with some general performance advice.

Performance advice

First of all, you don’t have to optimize for performance if you don’t have performance problems. If you do, you’re doing premature optimization. Instead, you start by writing readable code that you are comfortable with, after that you measure the performance, and if you have problems then you optimize.

It’s also better to spot the performance problems in the production build. That’s because the production code is much faster than the development build. As a result, a problem in development may be negligible in production. So don’t freak out with dropped frames in development.

Also, it’s good to add some CPU throttle in Chrome DevTools to emulate the performance on lower-end devices. Your development pc is probably quite powerful, so it will handle well whatever you throw at it.

Understanding that we have a problem

So having all that in mind, I started monitoring some interactions in my app with Chrome DevTools (performance tab). One problem I noticed (spoiler: they were more), was occurring when I clicked a card from the collection with the intention to add it into the deck. You can see the interaction in the following gif:

Add a card in the deck interaction.

First of all, I can hear the Hearthstone players saying: “Why do you even want to add Cursed Blade into your deck; that card is terrible.” I agree, but let’s move on. We see in the following image that the click handler took about 275ms with an x4 CPU throttle applied in the production build:

Production x4 slowdown

This is pretty bad if you have many users with mid to low-end devices. So another tip is to always use your analytics to see if it’s even worth spending your time optimizing. Maybe the majority of your users access your app from a desktop. I don’t have any users but whatever. This problem seemed interesting enough, so I continued investigating. The next thing we’ll do is to figure out if we use sub-optimally our 3rd party libraries.

Investigating 3rd party libraries

For this project, I’m using the FixedSizeGrid from react-window to display the cards. It’s similar to react-virtualized, but simpler, faster, and smaller. If you are not familiar with the library, you can skip to the next section; it’s not the point of the post anyway. I will now list the code for the component that renders the FixedSizeGrid and the code for the GridItem.

Take a quick look at the highlighted lines, and we’ll talk about it later:

Cards.jsx code snippet

Cards.jsx

import React, { useState, useEffect, useRef, useContext } from "react";
import PropTypes from "prop-types";
import styled from "styled-components";
import { FixedSizeGrid as Grid } from "react-window";
import AutoSizer from "react-virtualized-auto-sizer";
import memoize from "memoize-one";
import { boxShadow } from "styled-system";
import { toast } from "react-toastify";

import { Flex, Box, Heading, Centered, Text } from "../Primitives";
import GridItem from "./GridItem";
import { DeckContext } from "../../context/deck-context";
import CardDetailsModal from "../Modals/CardDetailsModal";

// skipping styling and boring utilities...


const createOpenModal = memoize((func) => (card) =>
  func({ open: true, card })
);
const createItemData = memoize(
  (columnCount, searchedCards, openModal) =>
    console.log("it runs!") || {
      columnCount,
      searchedCards,
      openModal,
    }
);

const Cards = ({ cards }) => {
  const { deck } = useContext(DeckContext); // skipping state and effects...
  return (
    <Container bg="tuscan" color="raisinBlack">
      {/* skipping... */}
      <Box height="100vh" style={{ overflowX: "hidden" }}>
        {searchedCards.length > 0 ? (
          <AutoSizer defaultWidth={1920} defaultHeight={1080}>
            {({ width, height }) => {
              const cardWidth = width < 500 || height < 500 ? 180 : 250;
              const cardHeight = width < 500 || height < 500 ? 250 : 345;

              const columnCount = Math.floor(width / cardWidth);
              const rowCount = Math.ceil(
                searchedCards.length / columnCount
              );

              const openModal = createOpenModal(setModalInfo);
              const itemData = createItemData(
                columnCount,
                searchedCards,
                openModal
              );
              return (
                <CenteredGrid
                  columnWidth={cardWidth}
                  columnCount={columnCount}
                  rowHeight={cardHeight}
                  rowCount={rowCount}
                  height={height}
                  width={width}
                  itemData={itemData}
                >
                  {GridItem}
                </CenteredGrid>
              );
            }}
          </AutoSizer>
        ) : (
          <Centered minHeight="50vh">
            <Text fontSize={4}>No cards. Try different filters.</Text>
          </Centered>
        )}
      </Box>
    </Container>
  );
};

Cards.propTypes = {
  cards: PropTypes.arrayOf(PropTypes.object).isRequired,
};

export default Cards;

GridItem.jsx code snippet

GridItem.jsx

import React, { useState, useEffect, useRef, useContext, memo } from "react";
import PropTypes from "prop-types";
import styled from "styled-components";
import { areEqual } from "react-window";
import { Image as BaseImage } from "rebass";
import { toast } from "react-toastify";

import { Box, Button } from "../Primitives";
import { FaSearch, FaMinus } from "../Icons";
import { addUnconditionally, validateAdd, inDeck } from "../../utils/deck";
import { DeckContext } from "../../context/deck-context";
import { addCard, removeCard } from "../../ducks/deck";

// ...skipping styling and utilities

const GridItem = memo(({ columnIndex, rowIndex, style, data }) => {
  const { columnCount, searchedCards, openModal } = data;
  const singleColumnIndex = columnIndex + rowIndex * columnCount;
  const cardToRender = searchedCards[singleColumnIndex];

  const errorToast = useRef();
  const imageRef = useRef();
  const { deck, dispatch } = useContext(DeckContext);

  const [quantity, setQuantity] = useState(
    cardToRender ? getQuantity(deck, cardToRender) : 2
  );
  useEffect(() => {
    setQuantity(getQuantity(deck, cardToRender));
  }, [cardToRender, deck]);

  return (
    <Container p={2} style={style}>
      {cardToRender && (
        <>
          <ActionContainer>
            {quantity && (
                /* ...skipping UI stuff */
              }
          </ActionContainer>
          <Button
            variant="transparent"
            aria-label={`Add ${cardToRender.name}`}
            data-microtip-position="bottom"
            role="tooltip"
            style={{ padding: 0 }}
            onClick={() => {
              try {
                const resultingDeck = addUnconditionally(deck, cardToRender);
                validateAdd(resultingDeck, deck.ignoreSize);
                dispatch(addCard({ card: cardToRender }));
              } catch (err) {
                if (!toast.isActive(errorToast.current)) {
                  errorToast.current = toast.error(`🔥 ${err.toString()}`);
                }
              }
            }}
          >
            <Image
              ref={imageRef}
              alt={cardToRender.name}
              style={{ opacity: getOpacity(cardToRender, quantity) }}
              src={`/resources/images/${cardToRender.imageUrl}`}
            />
          </Button>
        </>
      )}
    </Container>
  );
}, areEqual);

// ...skipping proptypes

export default GridItem;

In Cards.jsx I render a grid (CenteredGrid) and a GridItem as a child. I also use the DeckContext, and, as a result, the component renders when the deck in the context changes.

In GridItem.jsx, I again use the DeckContext, but, in addition to that, I use a click handler that dispatches to the reducer an action (addCard). Finally, I render an image and some other elements depending on the quantity. The quantity represents the number of copies of a card inside the deck.

Some other stuff I’ve highlighted are performance optimizations mentioned in this video by Brian Vaughn, the creator of the library. Let’s name those:

We avoid passing as a grid child an anonymous component. Instead, we pass a reference e.g. {GridItem} (Reference? Is that what it’s called? I’m not sure). Basically, if you pass an anonymous component, and the parent renders, it always mounts the children instead of updating them (ouch, that’s bad).
We’re using memo for the GridItem which is basically the equivalent of the PureComponent for functional components.
We are comparing the props of the GridItem with the areEqual function from react-window. This is the equivalent for shouldComponentUpdate that’s already implemented for Cell components of react-window grids.
We’re also using memoization for the props we pass with itemData (this is a Grid prop) in each GridItem from the Grid. Because some of the props we pass are complex types (not primitives), and we assign them at render, even if they are the same in value, they change in reference. A new object is created in every render, so the GridItem will render because it does only a shallow comparison of its props. So we memoize them, and if their value is the same, we take the cached reference.

P.S. about the memoization part: I think we could achieve the same with an areEqual function in the GridItem. Generally, you use memoization for expensive operations, and not so much for the reason we did here. Also, this explanation took really long, and this is generally a sign that I messed it up, so have that in mind…

This brought a significant improvement; from 550ms in development with an x4 CPU throttle to the current 439ms, that you’ll see shortly in the next section.

Using Chrome DevTools

Now that we know that we’re using our dependencies in the most performant way, let’s go in development mode, and investigate the issue further. The development mode has a really useful tool which is the User Timing API dropdown (named as Timings). You can also use the React Profiler if you have the React developer tools installed. The profiler is something similar to Timings dropdown, and I won’t use it here. If you’re reading this in 2020, you should use the new React profiler.

We narrow down to one of the red lines in the overview, marked by the horribly drawn red circle in the image below. A red line means that something took longer than expected, and as a result, we dropped some frames. We go to the main dropdown to find the click event, as we did in production (previous image). Now, because we are in development, we have access to the Timings dropdown. If we open it, we see the following:

Development x4 slowdown

We see that our AppContainer updates in 439ms. If we scroll down (next image), we see that we update the ParentFilterGroup, which is something like a navbar in the app, and the Cards component that takes the most time. In the Cards component, we see some components being updated. Those components are our GridItems:

Grid items take their sweet time to render

They take about 12ms to update, and one of them in the middle takes about 20ms to update. This is something that should not happen. We expected to see one GridItem updating for 20ms because this is the card we add to the deck, and it’s quantity changes. When the quantity changes, we render an extra button that, when clicked, removes the card from the deck, and we decrease the opacity of the image. The rest of the cards, though, should update almost instantly because nothing changes for them.

This happens because if we take look at the code for our GridItem, we can see that we’re using the DeckContext. The deck changes in the context when we add a card, so every GridItem renders again no matter what. Ideally, we would want to have the deck from DeckContext passed as a prop in the GridItem. This way, we can check with shouldComponentUpdate (or with an areEqual implementation in functional components) if the GridItem should re-render.

Splitting the GridItem component

To achieve that, we’ll have to split the GridItem component into 2 parts. We’ll end up with one parent component that uses the context, and a child component that takes the context data as props. We can see below the code after the changes:

Code after splitting:

Cards.jsx

// omitted...
<CenteredGrid
  columnWidth={cardWidth}
  columnCount={columnCount}
  rowHeight={cardHeight}
  rowCount={rowCount}
  height={height}
  width={width}
  itemData={itemData}
>
  {GridItemWithDeck}
</CenteredGrid>

// omitted...

GridItemWithDeck.jsx/GridItem.jsx

import React, { useRef, useContext, memo } from "react";
import PropTypes from "prop-types";
import styled from "styled-components";
import { areEqual } from "react-window";
import { Image as BaseImage } from "rebass";
import { toast } from "react-toastify";

// skipping internal dependencies and styling...

export const GridItemWithDeck = memo(
  ({ columnIndex, rowIndex, style, data }) => {
    const { columnCount, searchedCards, openModal } = data;
    const singleColumnIndex = columnIndex + rowIndex * columnCount;
    const cardToRender = searchedCards[singleColumnIndex];

    const { deck, dispatch } = useContext(DeckContext);
    const quantity = getQuantity(deck, cardToRender);
    return (
      <Container p={2} style={style}>
        <GridItem
          deck={deck}
          dispatch={dispatch}
          cardToRender={cardToRender}
          quantity={quantity}
          openModal={openModal}
        />
      </Container>
    );
  },
  areEqual
);

GridItemWithDeck.propTypes = {
  // skipping proptypes...
};

// So to recap:
// We don't render if the parent renders because
// we are a PureComponent (memo). We render if are props
// change and we determine that with a shallow
// comparison. Now in the case where the deck has
// changed because we added a card in the deck, we don't
// render if the quantity remains the same. We achieve
// the last by implementing shouldComponentUpdate (areEqual).
const childAreEqual = (prev, next) => {
  if (prev.deck !== next.deck && prev.quantity === next.quantity)
    return true;
  return false;
};
// eslint-disable-next-line react/no-multi-comp
const GridItem = memo(
  ({ deck, dispatch, cardToRender, openModal, quantity }) => {
    const errorToast = useRef();
    const imageRef = useRef();

    return cardToRender ? (
      <>
        <ActionContainer>
          {quantity && (
            <Button
              variant="icon"
              aria-label="Remove 1 card"
              data-microtip-position="bottom-left"
              role="tooltip"
              bg="paradisePink"
              onClick={() => {
                try {
                  dispatch(removeCard({ card: cardToRender }));
                } catch (err) {
                  if (!toast.isActive(errorToast.current)) {
                    errorToast.current = toast.error(
                      `🔥 ${err.toString()}`
                    );
                  }
                }
              }}
            >
              <FaMinus />
            </Button>
          )}
          <Button
            variant="icon"
            aria-label={`${cardToRender.name} details`}
            data-microtip-position="bottom-left"
            role="tooltip"
            bg={heroColors[cardToRender.cardClass]}
            onClick={() => openModal(cardToRender)}
          >
            <FaSearch />
          </Button>
        </ActionContainer>
        <Button
          variant="transparent"
          aria-label={`Add ${cardToRender.name}`}
          data-microtip-position="bottom"
          role="tooltip"
          style={{ padding: 0 }}
          onClick={() => {
            // skipping handler...
          }}
        >
          <Image
            ref={imageRef}
            alt={cardToRender.name}
            style={{ opacity: getOpacity(cardToRender, quantity) }}
            src={`/resources/images/${cardToRender.imageUrl}`}
          />
        </Button>
      </>
    ) : (
      <div />
    );
  },
  childAreEqual
);

GridItem.propTypes = {
  // skipping proptypes...
};
GridItem.defaultProps = {
  quantity: null,
};

export default GridItem;

In the code above, we now have 2 components: the GridItemWithDeck and the GridItem. The GridItemWithDeck uses the context, and it transforms the data from the Cards component into something more useful for the GridItem. It’s a memo (pure) component, and it checks its props on every update with the areEqual function from react-window.

The GridItem is also a memo component, but it checks its props with the custom childAreEqual function. That function basically says: “If the deck changed, but the quantity remains the same, don’t render”. This combined with the shallow prop comparison from memo should do the trick.

Let’s see how it fares when we try to measure it in development:

After splitting the grid items

283ms from 439ms. And if we zoom-in at the grid part:

A closer look into the Grid

We see that the one GridItem (card) that is added to the deck takes 32ms to update, and the rest much less at about 2ms. We’re happy with that because it’s the expected behavior.

One last thing we want to do is to build the code and go back to the production to measure. You’ll notice that a lot of back and forth goes on between development and production. These are the results in production with an x4 slowdown:

After splitting in production x4 slowdown

One of the longer handlers took 131ms to complete. I remind you that without the splitting the handler took about 275ms. If we test this without CPU throttling, we don’t drop any frames, so no red lines at all for desktop users.

Wait a minute… (PS)

So mission complete! Our app updates now almost twice as fast. We should celebrate, right?

NO! We messed up! (ok, I messed up, you didn’t do anything). We were not careful with the implementation of the areEqual function, and as a result, we don’t update the deck inside our click handlers. So, every time we add a card, we update only the deck reference of the card we just added. The rest of the cards remember the old deck. As a result, our validate function inside the click handler does nothing!

To fix this, we’ll have to make the button that adds the card into the deck a child of the GridItemWithDeck. We’ll lose some performance gains, but it’s necessary if we want an app without bugs. This is a performance test in development with x4 CPU throttle after the bug fix:

After fixing the bug in development x4 slowdown

The longest update took about 320ms after the final split.

As you can see, React has good reasons to re-render when a prop or a parent changes because it’s really easy to make mistakes when implementing the areEqual method…

What we’ve learned

You have to be careful with the context hook because when it changes, it forces expensive re-renders that may not be necessary.
You can mitigate that by splitting the components into 2 new components.
PureComponent (memo) and shouldComponentUpdate (areEqual) can help you skip renders.
We saw some performance optimizations for react-window that we can use when we render expensive items.
Spend your time optimizing for performance only if it matters to your users.
Always measure before you start optimizing.
We saw how to measure the performance of our app with Chrome DevTools and the performance tab.
Be careful when you implement the shouldComponentUpdate or areEqual methods because it’s easy to introduce bugs to your application.

Deep copy vs shallow copy vs assignment operator

Thu, 07 Mar 2019 22:12:03 GMT

In this post, we’ll try to understand the differences between deep object copy, shallow copy, and the assignment operator in JavaScript. To do that, we’ll have to talk about variables and values first.

Values in JavaScript

In this section, I will briefly talk about how I view the variables and values in JavaScript. There’s a lot to unpack, so if you want more details, check the resources below.

Variables point to values. Variables don’t point to other variables, and they don’t contain values. You can change where a variable points with the assignment operator (=) unless you declared it with the const keyword. In this case, you’ll get a syntax error.

In JavaScript, we have two kinds of values. The primitive values and the objects (some people may use the term reference value for object values). The primitive values can’t change (they are immutable) but the object values can (they are mutable). The primitive values include the string, number, bigint, boolean, null, undefined, and symbol types. Objects are a value type on their own. Inside the objects, you can find the arrays and the functions.

Resources

Values in JavaScript for impatient programmers., a book by Dr. Axel Rauschmayer.
Just JavaScript, an online course by Dan Abramov.

Assignment operator

Let’s now use the assignment operator to “copy” a value from a variable to another variable. Or, because I claimed that variables don’t contain values, let’s use the assignment operator to point both variables to the same value.

Primitives

Create a variable with the name message and point it to the string value "hello".

let message = "hello";

Create a variable with the name messageCopy. Point it to the same value the variable with the name message currently points to.

let message = "hello";
let messageCopy = message;

At this point, both variables point to the same string value "hello". If you log the variables in the console, you’ll see the "hello" string value two times:

let message = "hello";
let messageCopy = message;
console.log(message, messageCopy);

Because primitive values are immutable, if you try to change the first letter of the "hello world" string, nothing happens. It fails silently or throws a TypeError in strict mode.

let message = "hello";
let messageCopy = message;
messageCopy[0] = "H";
console.log(messageCopy);
// Prints "hello", not "Hello".

However, you can change where the messageCopy variable points to. The highlighted line points the messageCopy variable to the “hello world” string value:

let message = "hello";
let messageCopy = message;
messageCopy = "hello world";

The variable with the name message still points to the “hello” value, so if you log it in the console, the result will be “hello”.

let message = "hello";
let messageCopy = message;
messageCopy = "hello world";
console.log(message);

Objects (arrays)

We have a similar situation with array values. To remind you, JavaScript considers array values as object type values. You can point two variables to the same array value:

let numbers = [1, 2, 3];
let numbersCopy = numbers;
console.log(numbers, numbersCopy);
// Prints [1, 2, 3], [1, 2, 3]

You can change where the second variable points, without affecting the first (the same is true for primitive values):

let numbers = [1, 2, 3];
let numbersCopy = numbers;
numbersCopy = [4, 2, 3];
console.log(numbers, numbersCopy);
// Prints: [1, 2, 3], [4, 2, 3]

But because array values are mutable, you can change any element of the array value. In other words, you can mutate the array value, something you can’t do with primitive values like strings. In the following example, both variables point to the same array value, so both variables will see the change:

let numbers = [1, 2, 3];
let numbersCopy = numbers;
numbersCopy[0] = 4;
console.log(numbers, numbersCopy);
// Prints: [4, 2, 3], [4, 2, 3]

As a result, the assignment operator is not a good choice to “copy” an object value from one variable to another variable. If you mutate one value, both variables will see the change.

Shallow copy

Now, let’s see what happens if you try to make a shallow copy of an object. By shallow-copying an object, you create a new object value that has the same properties as the old object. Each property of the new object value points to the same values the old object’s properties are pointing. You’ll see what that means in a bit.

There are many ways you can shallow copy an object:

Let’s choose Object.assign for this example because it doesn’t require an external library:

let cat = {
  likes: "cuddles",
  favoriteHuman: { name: "Mary" },
};
let catCopy = Object.assign({}, cat);

The catCopy variable points to a mutable object value. That means you can change that object value if you want. Lets start by changing its likes property:

let cat = {
  likes: "cuddles",
  favoriteHuman: { name: "Mary" },
};
let catCopy = Object.assign({}, cat);

catCopy.likes = "food";

Nothing is stopping you from changing its object property:

let cat = {
  likes: "cuddles",
  favoriteHuman: { name: "Mary" },
};
let catCopy = Object.assign({}, cat);

catCopy.likes = "food";
cat.favoriteHuman.name = "John";

Let’s now log in the console the original object to see what’s going on:

let cat = {
  likes: "cuddles",
  favoriteHuman: { name: "Mary" },
};
let catCopy = Object.assign({}, cat);

catCopy.likes = "food";
cat.favoriteHuman.name = "John";
console.log(cat);

You see that the likes property remains intact, but the favoriteHuman property changed. This is probably something you didn’t intend to do. This will start to make sense if we see what the shallow copy does. First, it creates a new object value:

let catCopy = {
  /* ...*/
};

Then, it creates the properties the original object had:

let catCopy = {
  likes: "TODO",
  favoriteHuman: "TODO",
};

Finally, it points the new properties to the values the old properties were pointing to:

let catCopy = {
  likes: cat.likes,
  favoriteHuman: cat.favoriteHuman,
};

If you evaluate the cat.likes, you get the string value "cuddles". This is a string primitive value that you can’t mutate. If you evaluate the cat.favoriteHuman, you get the object value { name: "Mary" }. That object is not a new object but the same object for both the cat and the catCopy. So, this is the problem. Object values are mutable, so if you mutate the catCopy.favoriteHuman, the original object will also see the change in cat.favoriteHuman.

You can verify that both properties point to the same object with a strict equality check (===):

let cat = {
  likes: "cuddles",
  favoriteHuman: { name: "Mary" },
};
let catCopy = Object.assign({}, cat);

console.log(cat.favoriteHuman === catCopy.favoriteHuman);
// Prints: true

What you would probably want for the favoriteHuman is to point to a new object value. Something like this:

let catCopy = {
  likes: cat.likes,
  favoriteHuman: cat.favoriteHuman,
};

let catCopy = {
  likes: "cuddles",
  favoriteHuman: { name: "Mary" },
};

Now, you can mutate the catCopy object without worrying about the original cat object. This is what deep copy does.

Deep copy

Now let’s see what happens when you make a deep copy. Let’s use lodash.clonedeep for that:

import cloneDeep from "lodash.clonedeep";

let cat = {
  likes: "cuddles",
  favoriteHuman: { name: "Mary" },
};
let catCopy = cloneDeep(cat);

You can now mutate the properties of the new object:

import cloneDeep from "lodash.clonedeep";

let cat = {
  likes: "cuddles",
  favoriteHuman: { name: "Mary" },
};
let catCopy = cloneDeep(cat);
catCopy.likes = "food";
catCopy.favoriteHuman.name = "John";
console.log(cat);
// Prints:
// {
//   likes: 'cuddles',
//   favoriteHuman: { name: 'Mary' }
// }

As you can see, with a deep copy you can mutate the new object value without worrying about mutating the original object. The drawback is that a deep copy is more expensive in CPU cycles and memory.

Conclusion

If you don’t want to accidentally mutate your original objects:

Avoid passing around object values with the assignment operator, instead, use shallow copy.
If your objects are simple and have only primitive properties, you can shallow copy them and avoid performance overhead from deep copying.
If your objects are complex and have properties that point to other objects, you’ll have to deep copy them with a library like lodash.clonedeep.

Extra: higher-order functions

Somewhat related to the previous is the Array.prototype.map. With map, you take an array value and transform it into a new array value. You then assign that value to a new variable. You may think that because map creates a new array value, it’s safe to modify it. As it turns out, it depends on what type of value (primitive/object) you return from your transformer function.

In the following example, we have an array of cats (ok, 1 cat for brevity). We transform it into an array of people that are popular with cats.

let cats = [
  {
    likes: "cuddles",
    favoritePerson: { name: "Mary" },
  },
];
let peopleWithCats = cats.map((cat) => cat.favoritePerson);

// Mutate a person from the new array.
peopleWithCats[0].name = "John";

// Log in the console the original cat array.
console.log(cats);
// Result
// [
//  {
//    likes: "cuddles"
//    favoritePerson: { name: "John" },
//  }
// ]

The result shouldn’t be surprising because you return an existing object value from map. As a result, if you mutate the person object in the new array, you also mutate the person object inside the cat from the original cat array.

The same is true if you use the spread operator on the cat objects which performs a shallow copy. Notice that, in the following example, we create a copy of the cats array, not an array with people. We do that to illustrate the use of the spread operator:

let cats = [
  {
    likes: "cuddles",
    favoritePerson: { name: "Mary" },
  },
];
let catsCopy = cats.map((cat) => ({ ...cat }));

// Mutate a cat's favorite person from the new array.
catsCopy[0].favoritePerson.name = "John";

// Log in the console the original cat array.
console.log(cats);
// Result
// [
//  {
//    likes: "cuddles"
//    favoritePerson: { name: "John" },
//  }
// ]

A better way is to return a new object value inside map. In the next example, we create an array with people again:

let cats = [
  {
    likes: "cuddles",
    favoritePerson: { name: "Mary" },
  },
];
let peopleWithCats = cats.map(({ favoritePerson: { name } }) => ({
  // Name is a primitive value, so it's safe to return.
  name,
}));
// Or, because the people objects have only primitive properties,
// you can use the spread operator e.g. { ...favoritePerson }

// Mutate a person from the new array.
peopleWithCats[0].name = "John";

// Log in the console the original cat array.
console.log(cats);
// Result
// [
//  {
//    likes: "cuddles"
//    favoritePerson: { name: "Mary" },
//  }
// ]

As a result, you’ll have to know what value types you return from higher-order functions because it’s possible to mutate the original arrays.