Today I Learned
A collection of snippets, thoughts and notes about stuff I learned.
Code
Everything is available in a Git repository at github.com/badboy/til.
Summary
So far there are 35 TILs.
Android
- Set the date in the emulator - 2022-11-06
Set the date in the emulator
To set the date & time of the running system:
adb shell su root date 061604052021.00
The datetime is in the format:
MMDDhhmm[[CC]YY][.ss]
BigQuery
- WebAssembly in BigQuery - 2022-11-06
WebAssembly in BigQuery
So you can run WebAssembly code as part of a BigQuery SQL query.
Rust code:
#![allow(unused)] fn main() { #[no_mangle] extern "C" fn sum(a: i32, b: i32) -> i32 { a + b } }
Compiled using:
cargo build --target wasm32-unknown-unknown --release
with these compile settings in your Cargo.toml
:
[lib]
crate-type = ["cdylib"]
[profile.release]
opt-level = "s"
debug = false
lto = true
Turn the Wasm file into a C-like array:
xxd -i target/wasm32-unknown-unknown/release/add.wasm
Then drop the output into the below query:
CREATE TEMP FUNCTION sumInputs(x FLOAT64, y FLOAT64)
RETURNS FLOAT64
LANGUAGE js AS r"""
async function main() {
const memory = new WebAssembly.Memory({ initial: 256, maximum: 256 });
const env = {
'abortStackOverflow': _ => { throw new Error('overflow'); },
'table': new WebAssembly.Table({ initial: 0, maximum: 0, element: 'anyfunc' }),
'tableBase': 0,
'memory': memory,
'memoryBase': 1024,
'STACKTOP': 0,
'STACK_MAX': memory.buffer.byteLength,
};
const imports = { env };
const bytes = new Uint8Array([
0x00, 0x61, 0x73, 0x6d, 0x01, 0x00, 0x00, 0x00, 0x01, 0x07, 0x01, 0x60,
0x02, 0x7f, 0x7f, 0x01, 0x7f, 0x03, 0x02, 0x01, 0x00, 0x05, 0x03, 0x01,
0x00, 0x10, 0x06, 0x19, 0x03, 0x7f, 0x01, 0x41, 0x80, 0x80, 0xc0, 0x00,
0x0b, 0x7f, 0x00, 0x41, 0x80, 0x80, 0xc0, 0x00, 0x0b, 0x7f, 0x00, 0x41,
0x80, 0x80, 0xc0, 0x00, 0x0b, 0x07, 0x2b, 0x04, 0x06, 0x6d, 0x65, 0x6d,
0x6f, 0x72, 0x79, 0x02, 0x00, 0x03, 0x73, 0x75, 0x6d, 0x00, 0x00, 0x0a,
0x5f, 0x5f, 0x64, 0x61, 0x74, 0x61, 0x5f, 0x65, 0x6e, 0x64, 0x03, 0x01,
0x0b, 0x5f, 0x5f, 0x68, 0x65, 0x61, 0x70, 0x5f, 0x62, 0x61, 0x73, 0x65,
0x03, 0x02, 0x0a, 0x09, 0x01, 0x07, 0x00, 0x20, 0x01, 0x20, 0x00, 0x6a,
0x0b, 0x00, 0x0f, 0x0e, 0x2e, 0x64, 0x65, 0x62, 0x75, 0x67, 0x5f, 0x61,
0x72, 0x61, 0x6e, 0x67, 0x65, 0x73, 0x00, 0x21, 0x04, 0x6e, 0x61, 0x6d,
0x65, 0x01, 0x06, 0x01, 0x00, 0x03, 0x73, 0x75, 0x6d, 0x07, 0x12, 0x01,
0x00, 0x0f, 0x5f, 0x5f, 0x73, 0x74, 0x61, 0x63, 0x6b, 0x5f, 0x70, 0x6f,
0x69, 0x6e, 0x74, 0x65, 0x72, 0x00, 0x4d, 0x09, 0x70, 0x72, 0x6f, 0x64,
0x75, 0x63, 0x65, 0x72, 0x73, 0x02, 0x08, 0x6c, 0x61, 0x6e, 0x67, 0x75,
0x61, 0x67, 0x65, 0x01, 0x04, 0x52, 0x75, 0x73, 0x74, 0x00, 0x0c, 0x70,
0x72, 0x6f, 0x63, 0x65, 0x73, 0x73, 0x65, 0x64, 0x2d, 0x62, 0x79, 0x01,
0x05, 0x72, 0x75, 0x73, 0x74, 0x63, 0x1d, 0x31, 0x2e, 0x35, 0x32, 0x2e,
0x31, 0x20, 0x28, 0x39, 0x62, 0x63, 0x38, 0x63, 0x34, 0x32, 0x62, 0x62,
0x20, 0x32, 0x30, 0x32, 0x31, 0x2d, 0x30, 0x35, 0x2d, 0x30, 0x39, 0x29
]);
return WebAssembly.instantiate(bytes, imports).then(wa => {
const exports = wa.instance.exports;
const sum = exports.sum;
return sum(x, y);
});
}
return main();
""";
WITH numbers AS
(SELECT 1 AS x, 5 as y
UNION ALL
SELECT 2 AS x, 10 as y
UNION ALL
SELECT 3 as x, 15 as y)
SELECT x, y, sumInputs(x, y) as sum
FROM numbers;
References
- Running Python Code in BigQuery by Anna Scholtz
Dates
- The date of Easter - 2022-11-06
The date of Easter
def easter(year):
y = year;
c = y//100;
n = y-19*(y//19);
k = (c-17)//25;
i = c-c//4-(c-k)//3+19*n+15;
i = i-30*(i//30);
i = i-(i//28)*(1-(i//28)*(29//(i+1))*((21-n)//11));
j = y+y//4+i+2-c+c//4;
j = j-7*(j//7);
l = i-j;
m = 3+(l+40)//44;
d = l+28-31*(m//4);
return (m, d)
print(easter(2022)) # (4, 17)
Based on the wonderful explanation in §3. Calendrical. of the Inform7 documentation
Docker
- Docker on a remote host - 2022-11-06
- Run a shell with a Docker image - 2022-11-21
- SSH into the Docker VM on macOS - 2022-11-06
Docker on a remote host
docker context create remote --docker "host=ssh://hostname"
docker context use remote
Is the docker daemon running?
If the Docker daemon is not running on the remote host, you might see this error message:
Cannot connect to the Docker daemon at http://docker.example.com. Is the docker daemon running?
The docker.example.com
host is of course nonsense.
The solution: Start the Docker daemon on the remote host and it should work.
Run a shell with a Docker image
docker run -t -i --rm ubuntu:20.04 bash
Changing the platform, e.g. to use x86_64
when running on an M1 MacBook:
docker run -t -i --rm --platform linux/amd64 ubuntu:20.04 bash
Override the entrypoint:
docker run -t -i --rm --entrypoint /bin/bash ubuntu:20.04
SSH into the Docker VM on macOS
Run socat
first:
socat -d -d ~/Library/Containers/com.docker.docker/Data/debug-shell.sock pty,rawer
This will print some lines, including the PTY device opened, like
PTY is /dev/ttys029
Use that to connect using screen
:
screen /dev/ttys029
Firefox
- From build IDs to push log - 2022-06-23
From build IDs to push log
via @chutten:
Found a regression? Here's how to get a pushlog:
-
You have the build dates and you're gonna need revisions. Find the build before the regression and the build after the regression in this list: https://hg.mozilla.org/mozilla-central/firefoxreleases You want to record the Revision column someplace.
May 10 final f44e64a61ed1 May 11 final 61a83cc0b74b
-
Put the revisions in this template:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange={}&tochange={}
E.g. https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f44e64a61ed1&tochange=61a83cc0b74b
git
- Fixup commits - 2022-05-12
- Git helpers - 2022-05-13
- Last modification date of a file - 2021-06-15
- Rebase dependent branches with
--update-refs
- 2023-02-03
Fixup commits
Fixup commits are commits that build on top of an already existing commit. They can be squashed into the existing commit as a later fixup, e.g. to fix typos or formatting.
git commit
comes with builtin support for that: git commit --fixup=<commit>
,
where <commit>
is the existing commit to be modified.
See the documentation for details.
See also git helpers.
Git helpers
git-absorb
git commit --fixup, but automatic
See github.com/tummychow/git-absorb. See also Fixup commits.
git-revise
A handy tool for doing efficient in-memory commit rebases & fixups
See github.com/mystor/git-revise.
Last modification date of a file
Shows the date of the last commit that modified this file:
git log -1 --pretty="format:%ci" path/to/file
See PRETTY FORMATS
in git-log(1)
for all available formats.
Rebase dependent branches with --update-refs
To automatically adjust all intermediary branches of a larger patch stack rebase with --update-refs
on the latest commit:
git rebase -i main --autosquash --update-refs
GitHub
- GitHub Webhooks - 2022-11-06
GitHub Webhooks
GitHub can send webhooks to a configured server on events. By default this is done on any push event to the repository.
GitHub attaches an HMAC signature using the provided secret, which allows to verify that the content is really coming from GitHub. Documentation about this is available in Securing your webhooks.
In Rust one can verify the signature like this:
#![allow(unused)] fn main() { use hex::FromHex; use hmac::{Hmac, Mac, NewMac}; use sha2::Sha256; fn authenticate(key: &str, content: &[u8], signature: &str) -> bool { // https://developer.github.com/webhooks/securing/#validating-payloads-from-github const SIG_PREFIX: &str = "sha256="; let sans_prefix = signature[SIG_PREFIX.len()..].as_bytes(); match Vec::from_hex(sans_prefix) { Ok(sigbytes) => { let mut mac = HmacSha256::new_from_slice(key.as_bytes()).expect("HMAC can take key of any size"); mac.update(content); mac.verify(&sigbytes).is_ok() } _ => false, } } }
Gradle
- Run tests using Gradle - 2022-11-06
Run tests using Gradle
Run a single test:
./gradlew testDebugUnitTest --tests TestClassNameHere.testFunctionHere
Rerun tests when up-to-date
--rerun-tasks
or
test.outputs.upToDateWhen {false}
in the config
iOS
- iOS log output from device or simulator - 2022-12-07
- Trigger notifications in the simulator - 2022-11-03
iOS log output from device or simulator
On macOS: Use the Console
tool.
Alternatively: idevicesyslog
from libimobiledevice
.
Install:
brew install libimobiledevice
Usage:
idevicesyslog --process Client
Client
stands in for the process name (Client
is the name of Firefox iOS).
via https://bsddaemonorg.wordpress.com/2021/04/12/analysing-ios-apps-log-output/
Trigger notifications in the simulator
The notification goes into a notification.apns
file:
{
"aps": {
"alert": {
"title": "Push Notification",
"subtitle": "Test Push Notifications",
"body": "Testing Push Notifications on iOS Simulator"
}
}
}
Trigger the notification with:
xcrun simctl push booted com.apple.MobileSMS notification.apns
This sends it to the application com.apple.MobileSMS
.
Your own application's bundle identifier can be used if it handles notifications.
The application's bundle identifier can also be specified in the APNS file with the "Simulator Target Bundle"
key.
It can be left out on the command-line in that case.
APNS files can also be dropped on the simulator to be send.
Kotlin
var
vs.val
- Difference - 2022-11-06
var
vs. val
- Difference
Source: https://www.kotlintutorialblog.com/kotlin-var-vs-val/
var
: Mutable. Used to declare a mutable variable. It means the value of variable can be changed multiple times.val
: Immutable Used to declare a read only variable. It means once the value is assigned to variable, that can’t be changed later.
val
is same as final
in Java.
Linux
- Runing parallel tasks from make - 2022-11-06
- Symbols in shared libraries - 2022-11-06
Runing parallel tasks from make
With the combination of multiple tools, you can serve static files over HTTP and rerun a build step whenever any input file changes.
I use these tools:
With this Makefile
:
default:
$(MAKE) MAKEFLAGS=--jobs=2 dev
.PHONY: default
dev: serve rerun
.PHONY: dev
build:
# Put your build task here.
# I generate a book using https://github.com/rust-lang/mdBook
mdbook build
.PHONY: build
serve: build
@echo "Served on http://localhost:8000"
# Change to the generate build directory, then serve it.
cd _book && http
.PHONY: serve
rerun:
# fd respects your `.gitignore`
fd | entr -s 'make build'
.PHONY: rerun
All it takes to continously serve and build the project is:
make
Symbols in shared libraries
List all exported symbols of a dynamic library:
nm -gD path/to/libyourcode.so
To look at the largest objects/functions in libxul:
readelf -sW $NIGHTLY/libxul.so | sort -k 3 -g -r | head -n 100
To look at the disassembly:
objdump -dr $OBJ | c++filt
On macOS:
otool -tV $OBJ | c++filt
macOS
- List linked dynamic libraries - 2022-11-23
- Check who holds SecureInput lock - 2021-05-21
List linked dynamic libraries
otool -L path/to/liblib.dylib
Check who holds SecureInput lock
Individual applications on macOS can request SecureInput
mode, which disables some functionality that would otherwise allow to capture input.
One can check if SecureInput
is active and which process holds the lock:
$ ioreg -l -w 0 | grep SecureInput
| "IOConsoleUsers" = ({"kCGSSessionOnConsoleKey"=Yes,"kSCSecuritySessionID"=100024,"kCGSSessionSecureInputPID"=123,"kCGSSessionGroupIDKey"=20,
"kCGSSessionIDKey"=257,"kCGSessionLoginDoneKey"=Yes,"kCGSSessionSystemSafeBoot"=No,"kCGSSessionUserNameKey"="user",
"kCGSessionLongUserNameKey"="username","kCGSSessionAuditIDKey"=100001,"kCGSSessionLoginwindowSafeLogin"=No,"kCGSSessionUserIDKey"=101})
The kCGSSessionSecureInputPID
holds the PID of the process that holds the SecureInput
lock.
Find that process with ps
:
ps aux | grep $pid
nginx
- Match and act on query parameters - 2022-12-20
Match and act on query parameters
In order to serve different files based on query parameters, first create a map:
map $query_string $resource_name {
~resource=alice$ alice;
~resource=bob$ bob;
}
Then in your server
block match the location and inside rewrite the URL the way you need it:
location = /.well-known/thing {
root /var/www/thing;
if ($resource_name) {
rewrite ^(.*)$ /$resource_name.json break;
}
try_files $uri = 404;
}
Now if someone requests /.well-known/thing?resource=alice
nginx will serve /var/www/thing/alice.json
.
PostgreSQL
- Meta commands in
psql
- 2022-11-06
Meta commands in psql
Command | Note |
---|---|
\l | List databases |
\c | Connect to database |
\dt | List tables |
\d $table | List schema of $table |
Proxmox
- Resize disks of a VM - 2022-11-06
Resize disks of a VM
Source: https://pve.proxmox.com/wiki/Resize_disks
- Extend the disk in the web UI
- Run
partprobe
on the machine - Run
parted
print
to show current layout. This will ask you to fix the GPT. Say "Fix"resizepart 1
--1
being the partition ID- It asks for the end. Type
100%
- Resize the filesystem:
resize2fs /dev/vda1
Python
- Modify integer literals - 2022-11-06
- Deduplicate a list and keep the order - 2023-03-30
- pip - Install from Git - 2022-11-06
- Strip Markdown syntax - 2022-11-06
Modify integer literals
Integer literals in Python refer to the same object every time they are used. One can modify those objects:
from sys import getsizeof
from ctypes import POINTER, c_void_p, c_char, cast
def read_int(obj: int, vv=True) -> bytes:
size = getsizeof(obj)
ptr = cast(c_void_p(id(obj)), POINTER(c_char))
buf = ptr[0:size]
if vv:
print(f"int obj @ {hex(id(obj))}: {buf.hex(' ')}")
return buf
def write_int(dst: int, src: int):
raw_src = read_int(src, False)
dst_ptr = cast(c_void_p(id(dst)), POINTER(c_char))
for (idx, c) in enumerate(raw_src):
dst_ptr[idx] = c
read_int(1)
write_int(1, 2)
read_int(1)
a = 1
b = 2
print(a + b)
(via https://twitter.com/segfault_witch/status/1512160978129068032)
Deduplicate a list and keep the order
duplicated_list = [1,1,2,1,3,4,1,2,3,4]
ordered = list(dict.fromkeys(duplicated_list)) # => [1, 2, 3, 4]
pip - Install from Git
To install a Python package from Git instead of a PyPi-released version do this:
pip install git+ssh://git@github.com/account/repository@branch#egg=package-name
See also: Useful tricks with pip install URL and GitHub
Strip Markdown syntax
In order to strip Markdown syntax and leave only the plain text output one can patch the Markdown parser:
from markdown import Markdown
from io import StringIO
def unmark_element(element, stream=None):
if stream is None:
stream = StringIO()
if element.text:
stream.write(element.text)
for sub in element:
unmark_element(sub, stream)
if element.tail:
stream.write(element.tail)
return stream.getvalue()
Markdown.output_formats["plain"] = unmark_element
__md = Markdown(output_format="plain")
__md.stripTopLevelTags = False
def strip_markdown(text):
return __md.convert(text)
Then call the strip_markdown
function:
text = """
# Hello *World*!
[Today I learned](https://fnordig.de/til)
"""
print(strip_markdown(text))
This results in:
Hello World!
Today I learned
(via https://stackoverflow.com/a/54923798)
Rust
- No-op allocator - 2022-11-06
- Not-equal types - 2022-11-06
- Random values using only libstd - 2022-11-06
- Testing code blocks in the README - 2022-11-06
No-op allocator
use std::alloc::{Layout, GlobalAlloc}; #[global_allocator] static ALLOCATOR: NoopAlloc = NoopAlloc; struct NoopAlloc; unsafe impl GlobalAlloc for NoopAlloc { unsafe fn alloc(&self, _layout: Layout) -> *mut u8 { std::ptr::null_mut() } unsafe fn dealloc(&self, _ptr: *mut u8, _layout: Layout) {} } #[no_mangle] pub extern "C" fn add(left: i32, right: i32) -> i32 { left + right } fn main() {}
This also reduces the generated wat (WebAssembly text format) to a short and readable output:
(module
(type $t0 (func (param i32 i32) (result i32)))
(func $add (export "add") (type $t0) (param $p0 i32) (param $p1 i32) (result i32)
(i32.add
(local.get $p1)
(local.get $p0)))
(func $main (export "main") (type $t0) (param $p0 i32) (param $p1 i32) (result i32)
(unreachable)
(unreachable))
(memory $memory (export "memory") 16)
(global $__data_end (export "__data_end") i32 (i32.const 1048576))
(global $__heap_base (export "__heap_base") i32 (i32.const 1048576)))
Not-equal types
// requires nightly! #![feature(auto_traits)] #![feature(negative_impls)] use std::marker::PhantomData; auto trait NotSame {} impl<A> !NotSame for (A, A) {} struct Is<S, T>(PhantomData<(S,T)>); impl<S,T> Is<S,T> where (S,T): NotSame { fn absurd(&self) { } } fn main() { let t : Is<u32, u32> = Is(PhantomData); //t.absurd(); let z : Is<u32, i32> = Is(PhantomData); z.absurd(); }
Random values using only libstd
#![allow(unused)] fn main() { use std::collections::hash_map::RandomState; use std::hash::{BuildHasher, Hasher}; let random_value = RandomState::new().build_hasher().finish() as usize; println!("Random: {}", random_value); }
Testing code blocks in the README
via github.com/artichoke/intaglio
#![allow(unused)] fn main() { // Ensure code blocks in README.md compile #[cfg(doctest)] macro_rules! readme { ($x:expr) => { #[doc = $x] mod readme {} }; () => { readme!(include_str!("../README.md")); }; } #[cfg(doctest)] readme!(); }
SQLite
- Recursive Queries - 2022-12-04
- Temporary values in SQLite - 2022-11-06
- Working with dates - 2022-11-06
Recursive Queries
Generate a series of integers, per ID, with a given min
and max
.
CREATE TABLE data AS
WITH t (id, min, max) AS (
VALUES
(1, 4, 6),
(2, 6, 6),
(3, 7, 9)
)
SELECT * FROM t;
WITH RECURSIVE exp AS (
SELECT
id, min, max, min as x
FROM data
UNION ALL
SELECT
id, min, max, x+1 as x
FROM
exp
WHERE x < max
)
SELECT * from exp ORDER BY id;
Temporary values in SQLite
To select from some values:
WITH vals (k,v) AS (
VALUES
(0,-9999),
(1, 100)
)
SELECT * FROM vals;
To actually create a temporary table:
CREATE TEMP TABLE temp_table AS
WITH t (k, v) AS (
VALUES
(0, -99999),
(1, 100)
)
SELECT * FROM t;
Working with dates
Full docs: Date And Time Functions
Datetime of now
SELECT datetime('now');
Timestamp to datetime
SELECT datetime(1092941466, 'unixepoch');
Datetime to timestamp
SELECT strftime('%s', 'now');
- Exporting Twitter Spaces recording - 2022-11-06
Exporting Twitter Spaces recording
For better or for worse people are using Twitter Spaces more and more: audio-only conversations on Twitter. Simon Willison recently hosted one and wrote a TIL how to download it. I helped because it's actually easier than his initial solution, so I'm copying that here:
Exporting the recording using youtube-dl
Open the Twitter Spaces page, open your Firefox developer tools console in the network tab, filter for "m3u", then hit "Play" on the page. The network tab will capture the URL to the playlist file. Copy that.
Then use youtube-dl
(or one of its more recent forks like yt-dlp) to download the audio:
youtube-dl "https://prod-fastly-us-west-1.video.pscp.tv/Transcoding/v1/hls/GPI6dSzgZcfqRfMLplfNp_0xu1QXQ8iDEEA0KymUd5WuqOZCZ9LGGKY6vBQdumX7YV1TT2fGtMdXdl2qqtVvPA/non_transcode/us-west-1/periscope-replay-direct-prod-us-west-1-public/audio-space/playlist_16798763063413909336.m3u8?type=replay"
This will result in a .mp4
file (media container):
$ mediainfo "playlist_16798763063413909336 [playlist_16798763063413909336].mp4"
General
Complete name : playlist_16798763063413909336 [playlist_16798763063413909336].mp4
Format : ADTS
Format/Info : Audio Data Transport Stream
[...]
Audio
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
[...]
To extract only the audio part you can use ffmpeg:
ffmpeg -i "playlist_16798763063413909336 [playlist_16798763063413909336].mp4" -vn -acodec copy twitter-spaces-recording.aac