Mika Tuupola

Setup PX4 Drone From Scratch

Mon, 23 Mar 2026 00:00:00 +0000

Configuring a PX4 flight controller is more involved than iNav or Betaflight. Most commercial development is happening in PX4 because it has a permissive license. While I fly all three, PX4 has the most advanced features for autonomous flight.

The Goal

A quad that is flyable using both traditional RC transmitter (TX) and a Ground Control Station (GCS) using MAVLink protocol.

MAVLink protocol will be piggybacked over the ELRS radio. This removes the need to include additional telemetry radios. The GCS software will receive the MAVLink packets from the TX using a USB cable. This is more reliable than using the transmitter-provided WiFi network.

Hardware

The flight controller is a MicoAir743v2. It is a standard 30.5mm x 30.5mm mount controller and also has a build target for PX4.

I used the Nomad Gemini Xrossband ExpressLRS Module but any external module will do. You could fly without an external TX module, but it is required for connecting to the ground control station via USB.

Radio Setup

Using an external radio module requires you to change to External RF in the model setup.

Additionally there are two main things to configure in the transmitter.

MAVLink mode

Run the ExpressLRS script in the transmitter. You can find it from the SYS menu. Make sure packet rate is 500 2.4G and link mode is MAVLink. It should also automatically change the telemetry ratio to 1:2 (9921 bps).

Go to the other devices menu and also change the receiver RC protocol to MAVLink. Now they should talk MAVLink over ELRS with each other.

Flight modes

There will be seven different flight modes controlled by three physical switches: SA, SC and SB. Additionally the SF switch is used as panic recovery if your brain stops functioning. The idea is that left side has the simple flight modes. Moving to the right, complexity increases, and the rightmost switch is the panic button.

These are flight modes I personally prefer. I use the same switch layout and flight modes also with fixed wings.

Switch	Mode	Notes	Logical Switch
SA↑	Stabilized	Active when SB and SC are center (-)	L10
SA-	Acro	Active when SB and SC are center (-)	L11
SA↓	Manual	Active when SB and SC are center (-)	L12
SC↑	Mission	Overrides SA	L13
SC↓	Offboard	Overrides SA	L14
SB↑	Hold	Overrides SA and SC	L15
SB↓	RTH/RTL	Overrides SA and SC	L16
SF↓	Panic	Overrides SA, SC and SB	L17

PX4 firmware does not have a dedicated panic recovery. I currently use position hold as a kludgish alternative.

Killswitch

When first hitting a tree it might come as a surprise that unlike with iNav or Betaflight the PX disarm switch will not kill the motors. That is because disarming is allowed only after landing. Your drone will be hanging in the tree with propellers spinning and there is nothing you can do.

PX4 does have a killswitch but you cannot configure it to use the same channel as arming. My solution for this is switch panic and disarm at the same time.

Switch	Action	Notes	Logical Switch
SE↑	Disarm	Works only after landing
SE↓	Arm
SE↑ SF↓	Kill motors	Panic + Disarm == Kill, kill, kill	L19

This will not mess with my muscle memory because iNav and Betaflight setups will still disarm from the SE switch.

But how?

Configuring your radio to handle all this can be done in several ways. I chose the Programmer Way (tm) and used logical switches. The 0.1 second delay is there to avoid announcing a flight mode when SB or SC jumps over the center position.

You can configure this either by using the transmitter menus or by using the EdgeTX Companion software. I use the latter since it is much faster to use.

Logical switches are used to override the channel 6 output via mixes. This is what the flight controller reads to set the flight mode. Since offboard mode is not an actual flight mode in PX4, it is enabled using channel 9.

From the image you can also see channel 5 is used for arming, channel 7 for beeper and channel 8 for configuring the OSD.

The flight mode announcements are done using special functions tied to logical switches.

If you have a Jumper T15 you can download above settings from tuupola/edgetx-setups GitHub repository.

Firmware

Build

The default build of PX4 includes everything and the kitchen sink. I want to disable some features and also add support for MSP OSD. These instructions assume you have already set up the build environment.

$ git clone https://github.com/PX4/PX4-Autopilot.git --recursive
$ cd PX4-Autopilot
$ make micoair_h743-v2_default boardconfig

After changing the settings you can check the changes with git diff ..

$ git diff . 
diff --git a/boards/micoair/h743-v2/default.px4board b/boards/micoair/h743-v2/default.px4board
index 320f1d29e9..f828f2ad1c 100644
--- a/boards/micoair/h743-v2/default.px4board
+++ b/boards/micoair/h743-v2/default.px4board
@@ -33,7 +33,8 @@ CONFIG_DRIVERS_POWER_MONITOR_INA238=y
 CONFIG_DRIVERS_POWER_MONITOR_VOXLPM=y
 CONFIG_DRIVERS_PPS_CAPTURE=y
 CONFIG_DRIVERS_PWM_OUT=y
-CONFIG_COMMON_RC=y
+CONFIG_DRIVERS_RC_CRSF_RC=y
+CONFIG_DRIVERS_RC_INPUT=y
 CONFIG_COMMON_RPM=y
 CONFIG_DRIVERS_SMART_BATTERY_BATMON=y
 CONFIG_DRIVERS_TAP_ESC=y
@@ -54,7 +55,6 @@ CONFIG_MODULES_EVENTS=y
 CONFIG_MODULES_FLIGHT_MODE_MANAGER=y
 CONFIG_MODULES_FW_ATT_CONTROL=y
 CONFIG_MODULES_FW_AUTOTUNE_ATTITUDE_CONTROL=y
-CONFIG_MODULES_FW_POS_CONTROL=y
 CONFIG_MODULES_FW_RATE_CONTROL=y
 CONFIG_MODULES_GIMBAL=y
 CONFIG_MODULES_GYRO_CALIBRATION=y
@@ -105,3 +105,4 @@ CONFIG_SYSTEMCMDS_UORB=y
 CONFIG_SYSTEMCMDS_USB_CONNECTED=y
 CONFIG_SYSTEMCMDS_VER=y
 CONFIG_SYSTEMCMDS_WORK_QUEUE=y
+CONFIG_ARCH_CHIP_STM32H7=y

After the build finishes you can see how much memory is being used.

$ make micoair_h743-v2_default -j8
...
Memory region         Used Size  Region Size  %age Used
        ITCM_RAM:           0 B        64 KB      0.00%
           FLASH:     1765492 B      1792 KB     96.21%
       DTCM1_RAM:           0 B        64 KB      0.00%
       DTCM2_RAM:           0 B        64 KB      0.00%
        AXI_SRAM:       53684 B       512 KB     10.24%
           SRAM1:           0 B       128 KB      0.00%
           SRAM2:           0 B       128 KB      0.00%
           SRAM3:           0 B        32 KB      0.00%
           SRAM4:           0 B        64 KB      0.00%
          BKPRAM:           0 B         4 KB      0.00%

Flash

Flashing is best done using a USB cable.

$ make micoair_h743-v2_default upload
...
Bootloader version: PX4BLv1.15.4g
Sn: 003d001f3233510d33353834
Chip: 20036450
Family: STM32H7[4|5]x
Revision: V
Flash: 1966080 bytes
Windowed mode: False

Erase  : [====================] 100.0%
Program: [====================] 100.0%
Verify : [====================] 100.0%
Rebooting. Elapsed Time 25.219

You can now connect to the NuttX shell and configure the flight controller.

$ ./Tools/mavlink_shell.py /dev/ttyACM0
NuttShell (NSH) NuttX-11.0.0
nsh> 
nsh> ver all
HW arch: MICOAIR_H743_V2
PX4 git-hash: 24bbf5efd90335d7533f6769db7ee3d23a0e7aa1
PX4 version: 1.17.0 40 (17891392)
PX4 git-branch: main
OS: NuttX
OS version: Release 11.0.0 (184549631)
OS git-hash: 29450774d75a2ad4748ac93b2d8759f1ada48883
Build datetime: Mar 20 2026 17:48:19
Build uri: localhost
Build variant: default
Toolchain: GNU GCC, 13.2.1 20231009
PX4GUID: 000600000000333538343233510d003d001f
MCU: STM32H7[4|5]xxx, rev. V
...

Configure

RC Input

First thing to configure is the RC input. Since we are using MAVLink over ELRS, plug the receiver into TELEM1 and not the RC port where it usually goes.

Change RC input protocol to CRSF. The default is auto but explicitly setting it to the desired protocol is a safer option. Also change the TELEM1 serial baud to 460800 to match the RX when running in MAVLink mode. MAV_0 rate should be 9600.

These settings approximately match the transmitter which was configured for 500Hz packet rate resulting in 9921bps MAVLink rate.

nsh> param set RC_INPUT_PROTO 6
nsh> param set SER_TEL1_BAUD 460800
nsh> param set MAV_0_RATE 9600

RC Input Channels

Map the four primary flight control axes to RC channels in PX4. This is the standard AETR channel order.

nsh> param set RC_MAP_ROLL 1
nsh> param set RC_MAP_PITCH 2
nsh> param set RC_MAP_THROTTLE 3
nsh> param set RC_MAP_YAW 4
nsh> param save
nsh> reboot

Connect the flight controller to USB and start QGC to make sure channels are mapped correctly. You should see matching stick movement in the Radio tab.

If everything is ok click Calibrate button to calibrate the sticks. Make sure to also calibrate all the switches or you will have mysterious problems later.

Switch Channels

Set transmitter channels. Channel 5 for arming and channel 6 for flight mode.

nsh> param set RC_MAP_ARM_SW 5
nsh> param set RC_MAP_FLTMODE 6

Flight Modes

Flight modes are: stabilized, acro, manual, mission, position hold, and RTH. These match the radio setup explained above. Offboard mode is special. PX4 does not consider it a flight mode and instead activates the mode from a separate channel 9.

nsh> param set COM_FLTMODE1 8
nsh> param set COM_FLTMODE2 6
nsh> param set COM_FLTMODE3 0
nsh> param set COM_FLTMODE4 3
nsh> param set COM_FLTMODE5 4
nsh> param set COM_FLTMODE6 5
nsh> param set RC_MAP_OFFB_SW 9
nsh> param save
nsh> reboot

Make sure flight modes work as expected. You can listen to vehicle status in the NuttX shell. Look for changes in nav_state parameter when you flip the switches.

nsh> listener vehicle_status -r 1

Or use the QGC statusline.

Motors

Attach the battery. The motors will beep.

In the Actuators menu, set motor protocol to DShot300. With PX4 there really is no point in using DShot600. Then click Identify & Assign Motors and motors will spin one by one. Click on the image to confirm which motor was spinning.

Also make sure the motor is spinning in the correct direction. The direction should match the image in QGC. If not, you can either swap two motor wires or change it in the Actuators menu.

Motors won’t beep anymore. Now you are ready to calibrate.

Calibrate

Magnetometer calibration requires rotating the quad 360 degrees around every axis. While possible with USB, it will be cumbersome. MicoAir flight controller has Bluetooth support which makes the calibration process easier.

Configure Bluetooth

First, turn on Bluetooth and find the address of your flight controller.

$ bluetoothctl
[bluetooth]# power on
[bluetooth]# scan on
[bluetooth]# devices
Device 39:54:9E:52:09:D1 MicoAir743v2-78387

[bluetooth]# pair 39:54:9E:52:09:D1
[bluetooth]# trust 39:54:9E:52:09:D1

I was not able to make QGC connect directly to this Bluetooth device. Instead I had to make a virtual serial device which is bound to the Bluetooth device.

$ sudo rfcomm bind 0 39:54:9E:52:09:D1 1
$ stty -F /dev/rfcomm0                  
speed 57600 baud; line = 0;
min = 0; time = 0;
-brkint -icrnl -imaxbel
-opost
-isig -icanon -iexten -echo

In QGC, add a new serial connection Comm settings -> Add new link. I call it Bluetooth RFComm. You can now wirelessly connect to the FC with this connection.

Calibrate sensors

Connect to the Bluetooth RFComm and go to Sensors tab. Calibrate all the sensors one by one. Follow the instructions on the screen. Instructions are pretty self-explanatory.

After any major model rebuild you should calibrate the sensors again. Especially the magnetometer is susceptible to small changes in magnetic fields.

Test Flight

Before attempting autonomous flight, this would be the time to make a test flight. Make sure all the flight modes work. Triple check position hold and return to home really work. If not, double check magnetometer and flight controller orientations and redo calibration.

Configure Transmitter USB

Connect the external TX module to your computer using USB. In QGC, add a new serial connection. I call the connection Transmitter USB but any name will do.

You can now connect to the FC with this connection.

You can now configure the drone or even plan missions wirelessly. Congrats, the first autonomous flight is pretty exciting.

Additional Reading

See the ExpressLRS MAVLink documentation for a more in-depth explanation of how to configure MAVLink over ELRS. If you prefer a video, Painless360 has a great clip on the same subject.

Ultimate PX4 Tuning Guide by Chris Rosser for PID tuning, filter and everything else. This is an excellent resource.

See How to Calibrate a Magnetometer for theory and example code about how magnetometer calibration works.

Citizens' Initiatives Not So Private

Wed, 24 Dec 2025 00:00:00 +0000

“Liu Shaoqi subjected to public criticism” by Unknown Author is in the Public Domain, via Wikimedia Commons.

Finland’s leading daily newspaper Helsingin Sanomat recently wrote a hit piece against people who had supported citizens’ initiatives which the newspaper considers verboten. These included initiatives such as banning political strikes or Extinction Rebellion. One initiative was asking for budget cuts to YLE which is the Finnish equivalent of BBC with similar problems to the BBC.

For many, the phone call from a journalist and having their photo together with their employer published in the newspaper came as a surprise. After all the kansalaisaloite.fi service advertises itself like this.

“The system does not store the social security number of the person signing a statement of support. For each statement of support, an initiative specific oneway hash is calculated using the personal identity number, which securely prevents the same initiative from being supported multiple times. Based on this information, it is not possible to determine which different initiatives an individual person has supported in the service. The data of the statements of support are stored in the database strongly encrypted. The actions of the system’s administrators are recorded in a log.”

So how was the article possible? Let’s look at the code.

The Hash

The FAQ claims that service uses a oneway hash for storing the personal identity number ie. SSN. This seems to be true. The hash is created by first concatenating the initiative id, SSN and a salt. This concatenation is then hashed with SHA256. The hash is a binary so it is converted to a base64 string before saving to the database.

public String initiativeSupportHash(Long initiativeId, String ssn) {
    Assert.notNull(initiativeId, "initiativeId");
    Assert.hasText(ssn, "ssn");

    StringBuilder message = new StringBuilder(256)
    .append(initiativeId).append(MAC_DELIM)
    .append(ssn).append(MAC_DELIM)
    .append(password);

    return base64Encode(sha256(message.toString(), DEFAULT_ENCODING), DEFAULT_ENCODING);
}

So far so good. This prevents duplicate votes while keeping the actual SSN private. However, when you have access to the secret key, you could brute-force the SSN from the hashes. This would not take too long with modern hardware.

The Database

Looking at the schema there are three interesting columns: initiative_id, support_id and details.

create table support_vote (
    initiative_id bigint NOT NULL,
    -- base64(sha256(initiative_id & ssn & sharedSecret))
    support_id varchar(64) NOT NULL,
    -- aes( name & birthDate & municipality)
    details varchar(4096) NOT NULL,
    created timestamp NOT NULL DEFAULT now(),
    batch_id bigint,
    PRIMARY KEY (initiative_id, supportId),
    FOREIGN KEY (initiative_id) REFERENCES initiative(id),
    FOREIGN KEY (batch_id) REFERENCES support_vote_batch(id)
);

The initiative_id column is the initiative being supported. The hash from above is saved to support_id column. The most interesting part is the details column. It contains the full name, date of birth and home municipality of the person casting the vote.

private String buildVoteDetails(User user, Locale locale, DateTime now) {
    return new StringBuilder(128)
    // Last name
    .append(user.getLastName())
    .append(VRK_CSV_DELIM)

    // First names
    .append(user.getFirstNames())
    .append(VRK_CSV_DELIM)

    // DateOf Birth
    .append(user.getDateOfBirth().toString(VRK_DTF))
    .append(VRK_CSV_DELIM)

    // Home municipality
    .append(user.getHomeMunicipality().getTranslation(locale))
    .append(VRK_CSV_DELIM)

    // Voting time
    .append(now.toString(VRK_DTF))

    .toString();
}

At least it is encrypted with AES before saving to the database.

public EncryptionService(String password, String vetumaSharedSecret, int secureRandomReseedInterval, int encryptorPoolSize) {
    if (Strings.isNullOrEmpty(password)) {
        throw new IllegalArgumentException("registeredUserSecret was null or empty");
    }
    this.vetumaSharedSecret = vetumaSharedSecret;
    this.password = password;


    this.secureRandomReseedInterval = secureRandomReseedInterval;

    aesEncryptor = new PooledPBEStringEncryptor();
    aesEncryptor.setProvider(new BouncyCastleProvider());
    aesEncryptor.setAlgorithm(DEFAULT_ALGORITHM);
    aesEncryptor.setPassword(password);
    aesEncryptor.setKeyObtentionIterations(DEFAULT_KEY_OBTENTION_ITERATIONS);
    aesEncryptor.setPoolSize(encryptorPoolSize);
}

public String encrypt(String message) {
    // Add encryptor id to encrypted message
    return new StringBuilder(message.length() * 2)
    .append(DEFAULT_ENCRYPTOR_ID)
    .append(aesEncryptor.encrypt(message)).toString();
}

Observant readers have probably already spotted the problem.

No, it is not the fact that only one PBKDF iteration is used for key derivation. It is the fact users’ full name, date of birth and home municipality are saved in the same row in the same database table as the actual vote. You do not even need the SSN to find all the initiatives an individual person has supported in the service. You just query by the name and birthday.

Yeah, if nitpicking the sentence “Based on this information, it is not possible to determine which different initiatives an individual person has supported in the service” is technically correct since “this information” points to the SSN. It is still misleading because you can easily find the different initiatives supported by each individual.

Say what now?

To be fair the website also states the following.

“The names of the signatories of the statements of support are not published at any stage on the Kansalaisaloite.fi service.”

Which is technically correct since names have not been published on Kansalaisaloite.fi. Instead, they were published in Helsingin Sanomat.

“The organizers of a citizens’ initiative may only disclose information regarding the signatories to the Digital and Population Data Services Agency. If the Digital and Population Data Services Agency confirms that a citizens’ initiative has collected at least 50,000 statements of support, the Agency may disclose information from the statements; in other words, the information in its possession becomes public. If, according to the decision, there is an insufficient number of statements of support, the information will not become public.”

This still does not take away the fact that contrary to what is claimed, it is easy to find out all initiatives supported by each individual if you have access to the database.

It is worth noting also that the code in GitHub has not been updated since 2019. It is possible that development has continued behind closed doors and obvious shortcomings have been fixed in live code. If anyone knows where I can find the live codebase let me know.

Decentralized Social Media

Thu, 21 Jan 2021 00:05:00 +0200

Photo by Etienne Girardet on Unsplash

This is a post I have started to write several times already. First time 2017, then again in 2019 and 2020. I do not remember what was the drama then. Each time I procrastinated and thought things are not that bad yet.

Leaving all politics a side, this is not the way the Internet was supposed to be.

“The first truth is that the liberty of a democracy is not safe if the people tolerate the growth of private power to a point where it becomes stronger than their democratic state itself. That, in its essence, is fascism – ownership of government by an individual, by a group, or by any other controlling private power.”

—Franklin D. Roosevelt, excerpt from message to Congress, April 29, 1938

TL;DR: Social media services such as Twitter and Instagram could be recreated with an RSS like file format. No platform needed.

The Problem

Everything is centralized which gives the control of allowed ideas to the owner of the platform. If you deviate from the allowed thinking you will be silenced.

Bundesarchiv, Bild 183-1986-0813-454 / Franke, Klaus / CC-BY-SA 3.0, CC BY-SA 3.0 DE, via Wikimedia Commons

There have been attemps to create platforms which allow wider spectrum of opinion. These attempts are usually hindered by companies such as Apple and Google by removing the apps from their app stores.

Technically inclined Android users can install apps from alternative sources but even that does not help. If they don’t like you, companies such as Visa and Mastercard will not only blacklist the company but also the family members of the founders of the platform in order to stop you get funding for running your app.

If you are still able to get alternate sources of funding companies such as Amazon will stop providing you hosting services.

The Solution

No, the solution is not blockchain blockchain something. The solution is not having a new shiny platform either. Platforms are depedent on all the three things mentioned above.

Solution is to have an agreed file format and/or protocol for publishing the usual social media things such as posts, likes and reposts. Each user can publish this file anywhere they wish, as long as it is publicly accessible from the Internet.

Bundesarchiv, Bild 183-1990-0429-411 / Schindler, Karl-Heinz / CC-BY-SA 3.0, CC BY-SA 3.0 DE, via Wikimedia Commons

Simplest way to publish a feed would be to upload the single feed file to your own webserver. Alternatively it could be a GitHub Gist or a Pastebin paste. If you are worried about webhost cencorship IPFS might help. People preferring to stay anonymous could use onion services.

User who wants to follow you subscribes to the url of your feed with their client software. The client software then periodically checks for updates and renders a timeline view similar as Twitter or Instagram.

This is same as what RSS was for blogging world. Users decide what they subscribe to and what they see. There are no algorithms doing decisions on users behalf. Publishers can publish on any Internet connected server. Anyone can create a client software.

There is no single point of failure.

The File Format

I have given this some though. This might change when doing an actual implementation. Instead of XML I would use JSON which is lighter to parse. Goal is to be not bloated and easy to implement.

The title is your displayed username and the description is the tagline or what Instagram calls a bio.

The url is an array which contains one or more addresses where the client can access your feed. Only the first one is fetched by default. If the first one cannot be accessed next one in the array will be fetched instead. This is useful if you lose access to your primary feed you will not lose your subscribers.

Rest of the json contains your posts, reposts and likes.

{
    "title": "Mika Tuupola",
    "description": "Technology guy gone advertising",
    "url": [
        "https://www.appelsiini.net/social.json",
        "https://example.org/social.json"
    ],
    "posts": [],
    "reposts": [],
    "likes": []
}

Posts

Each post consists of an id, body, one or more media files and an optional ref. The id should be an instance of K-Sortable Unique IDentifier. KSUID was chosen because it has an embedded timestamp so there is no need for a separate timestamp in json data. KSUID can also be naturally sorted by their creation time which simplifies rendering the timeline.

The post content itself is in plaintext body property. Clients should be able to parse hashtags in the body. There probably should be some arbitrary character limit to discourage too long post. Maybe 1024 characters?

User can optionally post one or more images and / or videos in the media property. To enable single file publishing clients must be able to parse data urls. It is probably good idea to also support traditional urls.

If the post is a reply ref should contain the feed url and the id of the post being replied to. If post is not a reply ref should be null. Depending on settings client can use this to fetch the ref feed even if user does not currently subscribe to it.

{
    ...

    "posts": [
        {
            "id": "p6UEyCc8D8ecLijAI5zVwOTP3D0",
            "body": "Lorem ipsum dolor sit amet. #foo #bar",
            "media": [
                "data:image/png;base64,PAKBjRACCARjPAKBJggg==",
                "data:image/png;base64,FCKBjRAKCARjPAKAAgeg=="
            ],
            "ref": "https://example.com/social.json#1mr52fioahBqXD7HVgZ4IkKRrUf"

        },
        {
            "id": "0o5Fs0EELR0fUjHjbCnEtdUwQe3",
            "body": "Ut enim ad minim veniam. #foo",
            "media": [
                "https://example.com/foo.png"
            ],
            "ref": null
        }
    ]
}

Reposts

Repost happens when someone posts your post to their followers. Repost consists of id which is a KSUID and a ref to the post which is being posted. Thanks to KSUID simply sorting by the id puts the reposts in to correct location in the timeline.

{
    ...

    "reposts": [
        {
            "id": "1mr5Btzoz5G9M4Wl5ZZo9K4Y6oQ",
            "ref": "https://example.org/social.json#1mr5F1roAhBc9YFFftk6WNiZuQP"
        },
        {
            "id": "1mr5CRWwT8FRHrsq7Lqn3hJ3KV0",
            "ref": "https://example.org/social.json#1mr5HcxkKbNqs9X5oQ0fIfzENqP"
        },
        {
            "id": "1mr5DK70OQuTUPLvsCML2mkMHdB",
            "ref": "https://example.com/social.json#1mr5InVxCYH0dXSncivI667ESpb"
        }
    ]
}

Likes

To be honest I am not sure if this is needed. However seeing how popular such a feature is with now, why not. Same as repost, a like consists of id which is a KSUID and a ref to the post which is being liked.

{
    ...

    "likes": [
        {
            "id": "0uk1Hbc9dQ9pxyTqJ93IUrfhdGq",
            "ref": "https://example.org/social.json#p6UEyCc8D8ecLijAI5zVwOTP3D0"
        },
        {
            "id": "0uk1HdCJ6hUZKDgcxhpJwUl5ZEI",
            "ref": "https://example.org/social.json#1mr519x29wy6WAcgAftSb33dEgC"
        },
        {
            "id": "0uk1HcdvF0p8C20KtTfdRSB9XIm",
            "ref": "https://example.com/social.json#1mr52fioahBqXD7HVgZ4IkKRrUf"
        },
        {
            "id": "0uk1Ha7hGJ1Q9Xbnkt0yZgNwg3g",
            "ref": "https://example.org/social.json#1mr5448rXkn02LkyFXhOZNaCurh"
        }
    ]
}

The Problems

Discovery

Discovery means how can one find new people to subscribe to? I think this problem will fix itself in the same way the Internet has always fixed it. Primary discovery source will be reposts and likes. There will also most likely be people writing “Awesome People to Follow” style lists.

By design clients will index feeds with two degrees of separation. This means if you subscribe to feed A and that feed has a repost or a like to a feed B then your client will fully index both feeds A and B but will not index reposts or likes in feed B.

Clients can optionally have a setting to control the degrees of separation. If user sets this to 3 and the feed B has a repost from feed C the client software will index feeds A, B and C.

Posts indexed from feeds you do not directly follow can be used for rendering hashtags views and searches. There probably is a practical limit how many dedrees of separation is useful to index.

Identity

There is no usernames to reserve. Anyone can post anything under any name. How do people know you are who you are? Blogs have the same problem. Obvious proof of your identity would be your own domain. Maybe something could be done with PGP.

This is somehing I do not have a good answer to. However it is good to remember that even with usernames Twitter et al. have the problem of fake user accounts. In the end in the Internet you never know who the people really are.

Technical Barrier

Image by wal_172619 from Pixabay

Even if not especially high, the technical barrier for uploading the feed file to a webserver is probably too high for 80% of current social media users. The client software should enable uploading the feed file via ftp or scp. There is still problem of setting up the web server.

In the end there will most likely be multiple companies which offer hosting for the feed file. Same companies might offer their own reader clients. This will contradict with the requirement of not being dependent in a company. It is still better than being dependent on single social media platform though.

Programming the GD32V Longan Nano

Thu, 03 Dec 2020 00:00:00 +0000

RISC-V is gaining traction and some development boards have already popped up. One of them is the widely available Sipeed Longan Nano. Written information is a bit sparse at the moment. Let’s try to fix this with a quick writeup on wiring and programming the board. If you just want to see what the board can do here is a video instead.

Toolchain

The situation with RISC-V toolchain is somewhat confusing. There are RISC-V Software Collaboration, RISC-V Software, RISC-V Microcontroller and RISC-V repositories. Some vendors also provide their own prebuilt binaries but they often seem to be outdated.

Luckily the toolchain is easy to compile by yourself. Only downside is that the toolchain repository is huge and using a shallow copy did not seem to work.

$ git clone --recursive https://github.com/riscv-collab/riscv-gnu-toolchain.git
$ cd riscv-gnu-toolchain

$ mkdir /opt/riscv
$ ./configure --prefix=/opt/riscv --enable-multilib --with-cmodel=medany
$ make -j8
$ make install

Protip! If you also have Kendryte K210 based development boards, you can use the same toolchain if you configure it like this.

$ ./configure --prefix=/opt/riscv --enable-multilib --with-cmodel=medany \
  --with-arch=rv64imafc --with-abi=lp64f

Additionally you need the RISC-V version of OpenOCD. Below example is for Fedora. You might be missing different set of dependencies.

$ sudo dnf install libtool autoconf automake texinfo
$ sudo dnf install libusb-devel

$ git clone --recursive https://github.com/riscv/riscv-openocd.git
$ cd riscv-openocd

$ ./bootstrap nosubmodule
$ ./configure --prefix=/opt/riscv/
$ make -j8
$ make install

If you are using macOS you can also install the toolchain and OpenOCD with Homebrew.

$ brew tap riscv/riscv
$ brew install riscv-tools
$ brew install riscv-openocd

Nuclei SDK

For programming a GD32V series SoC best choice at the moment is the Nuclei SDK. It seems to be well maintained, exceptionally well structured and is quite easy to learn. Developing is done with your favourite text editor.

The SDK supports three different real time operating systems: FreeRTOS, UCOSII and RT-Thread. Since Longan Nano has only 32kB SRAM you might want to stay baremetal instead.

Nuclei SDK does support Longan Nano out of the box. Basic hello world and the Makefile would look like this.

#include <stdio.h>

void main(void)
{
    printf("Hello world\r\n");
}

TARGET = firmware
NUCLEI_SDK_ROOT = ../nuclei-sdk
SRCDIRS = .

include $(NUCLEI_SDK_ROOT)/Build/Makefile.base

You would compile and upload it with the following commands.

$ make SOC=gd32vf103 BOARD=gd32vf103c_longan_nano all
$ make SOC=gd32vf103 BOARD=gd32vf103c_longan_nano upload

The SDK will take care of basic things such use redirecting STDOUT to USART. This is where the Sipeed USB-JTAG/TTL RISC-V Debugger really pays off. In addition to the JTAG interface it also acts as an USB to TTL converter.

$ screen /dev/ttyUSB1 115200

Nuclei SDK Build Time: Nov 14 2020, 23:17:41
Download Mode: FLASHXIP
CPU Frequency 108540000 Hz
Hello world

Uploading with the Sipeed RISC-V debugger

To make both JTAG and serial interface work you need to connect all pins except NC (duh!) between the debugger and Longan Nano. If nitpicking second ground is also optional. Longan Nano RST is pin number 7 on the left side of the USB socket.

Debugger	Longan Nano
GND	GND
RXD	T0
TXD	R0
NC
GND	GND (optional)
TDI	JTDI
RST	RST
TMS	JTMS
TDO	JTDO
TCK	JTCK

When flashing you also need to connect the USB-C socket to provide power. When using Nuclei SDK you can flash the firmware with make.

$ make SOC=gd32vf103 BOARD=gd32vf103c_longan_nano upload

Uploading with the J-Link debugger

Debugger	Longan Nano
VREF	3v3
GND	GND
TDI	JTDI
NRST	RST
TMS	JTMS
TDO	JTDO
TCK	JTCK

You can also use SEGGER J-Link Commander to upload the firmware. The command line utility requires the firmare to be in hex format.

$ riscv64-unknown-elf-objcopy firmware.elf -O ihex firmware.hex

You can connect to Longan Nano’s JTAG interface automatically with the following.

$ JLinkExe -device GD32VF103VBT6 -speed 4000 -if JTAG \
  -jtagconf -1,-1 -autoconnect 1

SEGGER J-Link Commander V6.86e (Compiled Oct 16 2020 18:21:57)
DLL version V6.86e, compiled Oct 16 2020 18:21:45

...

J-Link>

To upload a new firmware manually, first halt the CPU and load the firmware.hex from above. Reset the core and peripherals. Set the program counter to 0x08000000 and finally enable the CPU and exit the command line utility.

J-Link>halt
J-Link>loadfile firmware.hex
J-Link>r
J-Link>setPC 0x08000000
J-Link>go
J-Link>q

If the new firmware does no run automatically you might need to powercycle the board.

While manually poking the internals might be fun it gets bothersome in the long run. You can also put the above commands to an external file and pass it to JLinkExe to do all of the above automatically.

$ cat upload.jlink
halt
loadfile firmware.hex
r
setPC 0x08000000
go
q

$ JLinkExe -device GD32VF103VBT6 -speed 4000 -if JTAG \
  -jtagconf -1,-1 -autoconnect 1 -CommanderScript upload.jlink

Uploading via USB

If you don’t have an external debugger it is also possible to upload via USB. At the time of writing you need to use latest dfu-util built from source.

$ git clone git://git.code.sf.net/p/dfu-util/dfu-util
$ cd dfu-util
$ ./autogen.sh
$ ./configure --prefix=/opt/dfu-util
$ make -j8
$ sudo make install

Then add /opt/dfu-util/bin to your $PATH and you should be able to flash the firmware via USB.

$ make SOC=gd32vf103 BOARD=gd32vf103c_longan_nano bin
$ dfu-util -d 28e9:0189 -a 0 --dfuse-address 0x08000000:leave -D firmware.bin

Before running dfu-util you need to put the board to download mode. Do this by holding down the RESET and BOOT buttons and then release them in the same order.

Uploading via Serial

Finally the GD32V also offers the good old serial bootloader. Although meant to be used with the STM32 family the stm32flash utility seems to work. You also need to add /opt/stm32flash/bin to your $PATH and you are good to go.

$ git clone https://git.code.sf.net/p/stm32flash/code stm32flash
$ cd stm32flash
$ autoreconf -i -v
$ ./configure --prefix=/opt/stm32flash
$ make -j8
$ sudo make install

You also need an USB to TTL converter. I am using TTL-234X-3V3. Note that while signal levels are 3V3 the VCC on this converter is 5V. With this converter VCC cannot be connected to the debug header. Connect it to the 5V pin instead. This will also power up the board.

USB to TTL	Longan Nano
GND	GND
VCC	5V
RXD	T0
TXD	R0

After wiring is correct you can test if the connection works. Go to bootloader mode by first holding down the RESET and BOOT buttons and then release them in same order. When in bootloader mode you can probe the usb device.

$ stm32flash /dev/ttyUSB0 -b 115200

stm32flash 0.5

http://stm32flash.sourceforge.net/

Interface serial_posix: 115200 8E1
GET returns unknown commands (0x 6)
Version      : 0x30
Option 1     : 0x00
Option 2     : 0x00
Device ID    : 0x0410 (STM32F10xxx Medium-density)
- RAM        : 20KiB  (512b reserved by bootloader)
- Flash      : 128KiB (size first sector: 4x1024)
- Option RAM : 16b
- System RAM : 2KiB

While stm32flash utility detects a device in it gets the specs wrong. Probably because officially it does not support the GD32V family. Flashing still works fine though. You can flash both .bin and .hex files.

$ stm32flash -g 0x08000000 -b 115200 -w firmware.bin /dev/ttyUSB0


stm32flash 0.5

http://stm32flash.sourceforge.net/

Using Parser : Raw BINARY
Interface serial_posix: 115200 8E1
GET returns unknown commands (0x 6)
Version      : 0x30
Option 1     : 0x00
Option 2     : 0x00
Device ID    : 0x0410 (STM32F10xxx Medium-density)
- RAM        : 20KiB  (512b reserved by bootloader)
- Flash      : 128KiB (size first sector: 4x1024)
- Option RAM : 16b
- System RAM : 2KiB
Write to memory
Erasing memory
Wrote address 0x08011d30 (100.00%) Done.

Starting execution at address 0x08000000... done.

Hello World on Screen

For graphics programming you could use HAGL. As the name implies HAGL is a hardware agnostic graphics library. To make it work with Longan Nano you also need a HAGL GD32V HAL

$ cd lib
$ git clone https://github.com/tuupola/hagl.git
$ git clone https://github.com/tuupola/hagl_gd32v_mipi.git hagl_hal

Add both dependencies to the project Makefile.

TARGET = firmware
NUCLEI_SDK_ROOT = ../nuclei-sdk
SRCDIRS = . lib/hagl/src lib/hagl_hal/src
INCDIRS = . lib/hagl/include lib/hagl_hal/include

include $(NUCLEI_SDK_ROOT)/Build/Makefile.base

With all this in place you can create a flashing RGB Hello world!

#include <nuclei_sdk_hal.h>
#include <hagl_hal.h>
#include <hagl.h>
#include <font6x9.h>

void main()
{
    color_t red = hagl_color(255, 0, 0);
    color_t green = hagl_color(0, 255, 0);
    color_t blue = hagl_color(0, 0, 255);

    hagl_init();
    hagl_clear_screen();

    while (1) {
        hagl_put_text(L"Hello world!", 48, 32, red, font6x9);
        delay_1ms(100);

        hagl_put_text(L"Hello world!", 48, 32, green, font6x9);
        delay_1ms(100);

        hagl_put_text(L"Hello world!", 48, 32, blue, font6x9);
        delay_1ms(100);
    };
}

Animations on screen

For testing purpose lets assume we have three functions to initialise, animate and render bouncing balls on the screen.

balls_init();
balls_animate();
balls_render();

You can find the actual implementation in GitHub. With above functions we can implement the main loop.

#include <hagl_hal.h>
#include <hagl.h>

void main()
{
    hagl_init();
    balls_init();

    while (1) {
        balls_animate();
        hagl_clear_screen();
        balls_render();
    };
}

Writing directly to display is fine for unanimated content. However above code will have a horrible flicker. Problem can be fixed by enabling double buffering in hte Makefile.

COMMON_FLAGS += -DHAGL_HAL_USE_DOUBLE_BUFFER

With double buffering enabled we also need to flush the contents from back buffer to display ie. front buffer. Here I also add some delay to slow down the animation.

#include <nuclei_sdk_hal.h>
#include <hagl_hal.h>
#include <hagl.h>

void main()
{
    hagl_init();
    balls_init();

    while (1) {
        balls_animate();
        hagl_clear_screen();
        balls_render();

        hagl_flush();
        delay_1ms(30);
    };
}

This is very naive approach and you need to manually adjust the delay to avoid tearing. It would be better to implement an fps limiter and flush the contents, for example 30 times per second.

Even though the Longan Nano is not the fastest and has only 32kB of memory the Nuclei SDK makes it pleasant to work with. Despite being tiny you can still do interesting stuff such as old school demo effects with it.

Additional reading

Creating a Videoplayer for ESP32

Mon, 25 May 2020 00:00:00 +0000

One of the things which makes embedded programming so interesting is that you are constantly dealing with restricted resources. Playing video files is quite resource heavy. Let’s see how ESP32 can handle this. If you have short attention span here is a video instead.

RAW RGB565

The easiest compression is no compression at all. Embedded displays are often configured to have RGB565 pixel format. This gives a good compromise between number of colors and bytes needed for one frame.

Raw video means each frame is consecutively stored in a file. There is no compression. There is no file header containing meta. Following code reads the first frame of such video.

To keeps the example code simple I am using an SD card helper component to initialise the SD card.

FILE *fp;
const size_t FRAME_SIZE = 320 * 180 * 2;
uint8_t *buffer = malloc(FRAME_SIZE);

sdcard_init();

fp = fopen("/sdcard/video.raw", "rb");
fread(buffer, FRAME_SIZE, 1, fp);
fclose(fp);

All frames can be read in a loop.

FILE *fp;
const size_t FRAME_SIZE = 320 * 180 * 2;
uint8_t *buffer = malloc(FRAME_SIZE);

fp = fopen("/sdcard/video.raw", "rb");

while (fread(buffer, FRAME_SIZE, 1, fp)) {
    printf("Read a frame.\n");
};

fclose(fp);

This is still not interesting since nothing is displayed. You could blit the frame one by one to the display. Better still the buffer could be a pointer directly to the framebuffer or back buffer.

I am using graphics library called HAGL. It gives you a pointer to the back buffer when initialising the display. The back buffer also has to be flushed to the display.

I am also adding an fps counter to see how well our video player is working.

FILE *fp;
const size_t FRAME_SIZE = 320 * 180 * 2;
bitmap_t *hagl;

hagl = hagl_init();
sdcard_init();

fp = fopen("/sdcard/video.raw", "rb");

while (fread(hagl->buffer, FRAME_SIZE, 1, fp)) {
    hagl_flush();
    printf("%.*f FPS\n", 1, fps());
};

fclose(fp);

Avoid the SD card bottleneck

The results are surprising bad. The video plays at only about 6 fps. The culprit seems to be newlibs default buffer size which causes files to be read in 128 byte chunks. Easy fix is to replace fread() with read().

Note that while fread() above returns number of frames read, read() here returns number of bytes read.

FILE *fp;
const size_t FRAME_SIZE = 320 * 180 * 2;
bitmap_t *hagl;

hagl = hagl_init();
sdcard_init();

fp = fopen("/sdcard/video.raw", "rb");

while (read(fileno(fp), hagl->buffer, FRAME_SIZE)) {
    hagl_flush();
    printf("%.*f FPS\n", 1, fps());
};

fclose(fp);

Results are now much better. The raw video can be played at approximately 10 fps. Consecutively calling printf() slows things down. Removing it would allow 12 fps playback. Things can still be sped up by moving the display flushing to second ESP32 core.

That would make the example code too complicated for this article so we are good for now. See the ESP32 videoplayer repository to see how it was done.

First part of this video shows the RGB565 player on a TTGO T4.

Motion JPEG

Motion JPEG or MJPEG is an intraframe video compression format. Each frame is compressed and stored separately as a JPG image. In laymans terms MJPEG is multiple JPG files concatenated into one bigger video file.

Uncompressing JPG images

MJPEG video is just a bunch of JPG files. If the device can uncompress JPG files it should be able to play MJPEG videos. Most embedded libraries I have seen piggypack on the Tiny JPEG Decompressor. It needs to be provided with two functions: one for reading a file and one for outputting the pixels.

Purpose of reader function is to either read bytes into buffer or skip bytes from the file pointer. If buffer is NULL function will fast forward the pointer size bytes.

uint16_t tjpgd_data_reader(JDEC *decoder, uint8_t *buffer, uint16_t size)
{
    FILE *fp = (FILE *)decoder->device;

    if (buffer) {
        /* Read bytes from input stream. */
        return read(fileno(fp), buffer, size);
    } else {
        /* Skip bytes from input stream. */
        if (lseek(fileno(fp), size, SEEK_CUR) > 0) {
            return size;
        }
        /* Seek failed, causes TJPGD to abort. */
        return 0;
    }
}

Writer function receives the uncompressed image in 8x8 blocks. Here I use HAGL provided blit function to blit that block into the display. You could also a putpixel function for the same purpose.

uint16_t tjpgd_data_writer(JDEC* decoder, void* bitmap, JRECT* rectangle)
{
    uint8_t width = (rectangle->right - rectangle->left) + 1;
    uint8_t height = (rectangle->bottom - rectangle->top) + 1;

    /* Create a HAGL bitmap from the uncompressed block. */
    bitmap_t block = {
        .width = width,
        .height = height,
        .depth = DISPLAY_DEPTH,
    };

    bitmap_init(&block, (uint8_t *)bitmap);

    /* Blit the block to the display. */
    hagl_blit(rectangle->left, rectangle->top + 30, &block);

    return 1;
}

Putting things together this is how you would display a JPG image.

uint8_t work[3100];
FILE *fp;
JDEC decoder;
JRESULT result;

sdcard_init();

fp = fopen("/sdcard/image.jpg", "rb");

result = jd_prepare(&decoder, tjpgd_data_reader, work, 3100, fp);
if (JDR_OK == result) {
    jd_decomp(&decoder, tjpgd_data_writer, 0);
};

hagl_flush();
fclose(fp);

Uncompressing MJPEG videos

So in theory to play an MJPEG file we should be able to just open the file and then read the JPG frames one by one.

uint8_t work[3100];
JDEC decoder;
JRESULT result;
FILE *fp;

hagl_init();
sdcard_init();

fp = fopen("/sdcard/video.mjpeg", "rb");

while (1) {
    /* Prepare next frame. */
    result = jd_prepare(&decoder, tjpgd_data_reader, work, 3100, fp);

    if (JDR_OK == result) {
        jd_decomp(&decoder, tjpgd_data_writer, 0);
        /* Frame ready, flush it to the display. */
        hagl_flush();
        printf("%.*f FPS\n", 1, fps());
    } else {
        printf("Prepare failed!\n");
    }
};

fclose(fp);

However there is a problem. It only reads the first frame and then spits out prepare failed errors. The problem here is the tjpgd_data_reader() function. It reads the compressed data in 512 byte blocks. This means most of the time file pointer is advanced past the EOI (end of image) marker into the territory of the next frame.

When reading the next frame file pointer is already past the image header and TJPGD fails to read the image meta. Simple solution here is to rewind the filepointer back to the next byte after EOI marker.

static uint16_t tjpgd_data_reader_new(JDEC *decoder, uint8_t *buffer, uint16_t size)
{
    FILE *fp = (FILE *)decoder->device;
    uint16_t bytes = 0;
    const uint16_t EOI = 0xffd9;
    uint8_t *match;

    if (buffer) {
        /* Read bytes from input stream. */
        bytes = read(fileno(fp), buffer, size);

        /* Search for EOI. */
        match = memmem(buffer, size, &EOI, 2);

        if (match) {
            /* Rewind back to previous EOI + 1. */
            int16_t offset = match - buffer;
            int16_t rewind = offset - size + 1;
            lseek(fileno(fp), rewind, SEEK_CUR);
            bytes += rewind;
        }

        return bytes;
    } else {
        /* Skip bytes from input stream. */
        if (lseek(fileno(fp), size, SEEK_CUR) > 0) {
            return size;
        }
        /* Seek failed, causes TJPGD to abort. */
        return 0;
    }
}

With this fix we now have a working MJPEG player. It is not the fastest. The above code is capable of playing 320x240 video with 6.5 fps. Again offloading display flushing to second ESP32 core would speed things up to 8 fps. See the ESP32 videoplayer repository to see how it was done.

Second part of this video starting at 00:45 shows the MJPEG player on a TTGO T4.

Additional reading

Hardware Agnostic Graphics Library for Embedded

Thu, 16 Apr 2020 00:00:00 +0000

What started as a project to learn C, I have now been using in all my hobby projects. HAGL is a hardware agnostic graphics library for embedded projects. It supports basic geometric primitives, bitmaps, blitting, fixed width fonts and an optional framebuffer.

Yet another graphics library?

Most embedded hobbyist projects do not seem to care about code reuse. Instead there is a graphics library for every different display driver for every different architecture. They all implement the same functions for drawing the graphical primitives This feels wrong. Graphics library should not know anything about the underlying hardware, responsibilities should be separated.

HAGL takes a different approach. It contains code only for drawing the primitives. It can be used with any microcontroller and display driver. It can also be used with normal computers. This is useful for testing your graphics code without need to flash it to the microcontroller.

Hardware abstraction layer

To use Copepod with your microcontroller and display driver you must provide a HAL. The only mandatory function for HAL to provide is for putting a single pixel on the display. Copepod will use the putpixel funtion to draw all the other graphical primitives. For examples of this see libgd HAL and libsdl2 HAL.

For improved speed the HAL can also provide accelerated functions for bitmap blitting and horizontal and vertical lines. See ESP MIPI DCS HAL for an example of a hardware accelerated HAL. This is also the one I use with my ESP32 projects. It supports most of the displays hobbyists currently use.

How fast is it?

Speed mostly depends on two things. Everything is much faster when double buffering is enabled. Also the HAL implementation dictates a lot. I have been testing with the TTGO T-Display, TTGO T4, M5StickC and M5Stack.

NOTE! Links above are affiliate links. If you buy something I will be a happy puppy.

In the below table numbers are operations per second with double buffering. ESP32 is clocked at the default 160MHz. Bigger number is better. T-Display and M5StickC have higher numbers because they have smaller resolution. Smaller resolution means less bytes to push to the display.

	T4	T-Display	M5Stack	M5StickC
hagl_put_pixel()	304400	304585	340850	317094
hagl_draw_line()	10485	14942	12145	31293
hagl_draw_circle()	15784	16430	17730	18928
hagl_fill_circle()	8712	9344	9982	13910
hagl_draw_ellipse()	8187	8642	9168	10019
hagl_fill_ellipse()	3132	3457	3605	5590
hagl_draw_triangle()	3581	5137	4160	11186
hagl_fill_triangle()	1246	1993	1654	6119
hagl_draw_rectangle()	22759	30174	26910	64259
hagl_fill_rectangle()	2191	4849	2487	16146
hagl_draw_rounded_rectangle()	17660	21993	20736	39102
hagl_fill_rounded_rectangle()	2059	4446	2313	13270
hagl_draw_polygon()	2155	3096	2494	6763
hagl_fill_polygon()	692	1081	938	3295
hagl_put_char()	29457	29131	32429	27569
hagl_flush()	32	76	32	96

When double buffering is disabled everything is much slower. On the positive side you save lots of memory.

	T4	T-Display	M5Stack	M5StickC
hagl_put_pixel()	16041	15252	16044	24067
hagl_draw_line()	113	172	112	289
hagl_draw_circle()	148	173	145	230
hagl_fill_circle()	264	278	261	341
hagl_draw_ellipse()	84	103	85	179
hagl_fill_ellipse()	114	128	116	191
hagl_draw_triangle()	37	54	37	114
hagl_fill_triangle()	72	111	72	371
hagl_draw_rectangle()	2378	2481	2374	3482
hagl_fill_rectangle()	91	146	91	454
hagl_draw_rounded_rectangle()	458	535	459	808
hagl_fill_rounded_rectangle()	87	139	79	400
hagl_draw_polygon()	21	33	19	71
hagl_fill_polygon()	43	66	49	228
hagl_put_char)	4957	4264	4440	2474
hagl_flush()	x	x	x	x

You can run the speed tests yourself by checking out the speedtest repository.

Graphical functions

The function calls themselves should be pretty self explanatory. Most of them take coordinates and a RGB565 color. Out of bounds coordinates are clipped to the current display.

Put a pixel

for (uint32_t i = 1; i < 100000; i++) {
    int16_t x0 = rand() % DISPLAY_WIDTH;
    int16_t y0 = rand() % DISPLAY_HEIGHT;
    color_t color = rand() % 0xffff;

    hagl_put_pixel(x0, y0, color);
}

Draw a line

for (uint16_t i = 1; i < 1000; i++) {
    int16_t x0 = rand() % DISPLAY_WIDTH;
    int16_t y0 = rand() % DISPLAY_HEIGHT;
    int16_t x1 = rand() % DISPLAY_WIDTH;
    int16_t y1 = rand() % DISPLAY_HEIGHT;
    color_t color = rand() % 0xffff;

    hagl_draw_line(x0, y0, x1, y1, color);
}

Draw a horizontal line

for (uint16_t i = 1; i < 1000; i++) {
    int16_t x0 = rand() % (DISPLAY_WIDTH / 2);
    int16_t y0 = rand() % DISPLAY_HEIGHT;
    int16_t width = rand() % (DISPLAY_WIDTH - x0);
    color_t color = rand() % 0xffff;

    hagl_draw_hline(x0, y0, width, color);
}

Draw a vertical line

for (uint16_t i = 1; i < 1000; i++) {
    int16_t x0 = rand() % DISPLAY_WIDTH;
    int16_t y0 = rand() % (DISPLAY_HEIGHT / 2);
    int16_t height = rand() % (DISPLAY_HEIGHT - y0);
    color_t color = rand() % 0xffff;

    hagl_draw_vline(x0, y0, height, color);
}

Draw a circle

for (uint16_t i = 1; i < 500; i++) {
    int16_t x0 = DISPLAY_WIDTH / 2;
    int16_t y0 = DISPLAY_HEIGHT / 2;
    int16_t radius = rand() % DISPLAY_WIDTH;
    color_t color = rand() % 0xffff;

    hagl_draw_circle(x0, y0, radius, color);
}

Draw a filled circle

for (uint16_t i = 1; i < 500; i++) {
    int16_t x0 = rand() % DISPLAY_WIDTH;
    int16_t y0 = rand() % DISPLAY_HEIGHT;
    int16_t radius = rand() % 100;
    color_t color = rand() % 0xffff;

    hagl_fill_circle(x0, y0, radius, color);
}

Draw an ellipse

for (uint16_t i = 1; i < 500; i++) {
    int16_t x0 = DISPLAY_WIDTH / 2;
    int16_t y0 = DISPLAY_HEIGHT / 2;
    int16_t rx = rand() % DISPLAY_WIDTH;
    int16_t ry = rand() % DISPLAY_HEIGHT;
    color_t color = rand() % 0xffff;

    hagl_draw_ellipse(x0, y0, rx, ry, color);
}

Draw a filled ellipse

for (uint16_t i = 1; i < 500; i++) {
    int16_t x0 = rand() % DISPLAY_WIDTH;
    int16_t y0 = rand() % DISPLAY_HEIGHT;
    int16_t rx = rand() % DISPLAY_WIDTH / 4;
    int16_t ry = rand() % DISPLAY_HEIGHT / 4;
    color_t color = rand() % 0xffff;

    hagl_draw_ellipse(x0, y0, rx, ry, color);
}

Draw a triangle

int16_t x0 = rand() % DISPLAY_WIDTH;
int16_t y0 = rand() % DISPLAY_HEIGHT;
int16_t x1 = rand() % DISPLAY_WIDTH;
int16_t y1 = rand() % DISPLAY_HEIGHT;
int16_t x2 = rand() % DISPLAY_WIDTH;
int16_t y2 = rand() % DISPLAY_HEIGHT;
color_t color = rand() % 0xffff;

hagl_draw_triangle(x0, y0, x1, y1, x2, y2, color);

Draw a filled triangle

int16_t x0 = rand() % DISPLAY_WIDTH;
int16_t y0 = rand() % DISPLAY_HEIGHT;
int16_t x1 = rand() % DISPLAY_WIDTH;
int16_t y1 = rand() % DISPLAY_HEIGHT;
int16_t x2 = rand() % DISPLAY_WIDTH;
int16_t y2 = rand() % DISPLAY_HEIGHT;
color_t color = rand() % 0xffff;

hagl_fill_triangle(x0, y0, x1, y1, x2, y2, color);

Draw a rectangle

for (uint16_t i = 1; i < 50; i++) {
    int16_t x0 = rand() % DISPLAY_WIDTH;
    int16_t y0 = rand() % DISPLAY_HEIGHT;
    int16_t x1 = rand() % DISPLAY_WIDTH;
    int16_t y1 = rand() % DISPLAY_HEIGHT;
    color_t color = rand() % 0xffff;

    hagl_draw_rectangle(x0, y0, x1, y1, color);
}

Draw a filled rectangle

for (uint16_t i = 1; i < 10; i++) {
    int16_t x0 = rand() % DISPLAY_WIDTH;
    int16_t y0 = rand() % DISPLAY_HEIGHT;
    int16_t x1 = rand() % DISPLAY_WIDTH;
    int16_t y1 = rand() % DISPLAY_HEIGHT;
    color_t color = rand() % 0xffff;

    hagl_fill_rectangle(x0, y0, x1, y1, color);
}

Draw a rounded rectangle

for (uint16_t i = 1; i < 30; i++) {
    int16_t x0 = rand() % DISPLAY_WIDTH;
    int16_t y0 = rand() % DISPLAY_HEIGHT;
    int16_t x1 = rand() % DISPLAY_WIDTH;
    int16_t y1 = rand() % DISPLAY_HEIGHT;
    int16_t r = 10
    color_t color = rand() % 0xffff;

    hagl_draw_rounded_rectangle(x0, y0, x1, y1, r, color);
}

Draw a filled rounded rectangle

for (uint16_t i = 1; i < 30; i++) {
    int16_t x0 = rand() % DISPLAY_WIDTH;
    int16_t y0 = rand() % DISPLAY_HEIGHT;
    int16_t x1 = rand() % DISPLAY_WIDTH;
    int16_t y1 = rand() % DISPLAY_HEIGHT;
    int16_t r = 10
    color_t color = rand() % 0xffff;

    hagl_fill_rounded_rectangle(x0, y0, x1, y1, r, color);
}

Draw a polygon

You can draw polygons with unlimited number of vertices which are passed as an array. Pass the number of vertices as the first argument.

int16_t x0 = rand() % DISPLAY_WIDTH;
int16_t y0 = rand() % DISPLAY_HEIGHT;
int16_t x1 = rand() % DISPLAY_WIDTH;
int16_t y1 = rand() % DISPLAY_HEIGHT;
int16_t x2 = rand() % DISPLAY_WIDTH;
int16_t y2 = rand() % DISPLAY_HEIGHT;
int16_t x3 = rand() % DISPLAY_WIDTH;
int16_t y3 = rand() % DISPLAY_HEIGHT;
int16_t x4 = rand() % DISPLAY_WIDTH;
int16_t y4 = rand() % DISPLAY_HEIGHT;
color_t color = rand() % 0xffff;
int16_t vertices[10] = {x0, y0, x1, y1, x2, y2, x3, y3, x4, y4};

hagl_draw_polygon(5, vertices, color);

Draw a filled polygon

You can draw filled polygons with up to 64 vertices which are passed as an array. First argument is the number of vertices. Polygon does not have to be concave.

int16_t x0 = rand() % DISPLAY_WIDTH;
int16_t y0 = rand() % DISPLAY_HEIGHT;
int16_t x1 = rand() % DISPLAY_WIDTH;
int16_t y1 = rand() % DISPLAY_HEIGHT;
int16_t x2 = rand() % DISPLAY_WIDTH;
int16_t y2 = rand() % DISPLAY_HEIGHT;
int16_t x3 = rand() % DISPLAY_WIDTH;
int16_t y3 = rand() % DISPLAY_HEIGHT;
int16_t x4 = rand() % DISPLAY_WIDTH;
int16_t y4 = rand() % DISPLAY_HEIGHT;
color_t color = rand() % 0xffff;
int16_t vertices[10] = {x0, y0, x1, y1, x2, y2, x3, y3, x4, y4};

hagl_fill_polygon(5, vertices, color);

The library supports Unicode fonts in fontx format. It only includes three fonts by default. You can find more at tuupola/fonts repository.

Put a single char

for (uint16_t i = 1; i < 10000; i++) {
    int16_t x0 = rand() % DISPLAY_WIDTH;
    int16_t y0 = rand() % DISPLAY_HEIGHT;
    color_t color = rand() % 0xffff;
    char ascii = rand() % 127;

    hagl_put_char(ascii, x0, y0, color, font8x8);
}

Put a string

The library supports Unicode fonts in fontx format. It only includes three fonts by default. You can find more at tuupola/fonts repository.

for (uint16_t i = 1; i < 10000; i++) {
    int16_t x0 = rand() % DISPLAY_WIDTH;
    int16_t y0 = rand() % DISPLAY_HEIGHT;
    color_t color = rand() % 0xffff;

    hagl_put_text("YO! MTV raps.", x0, y0, color, font8x8);
}

Additional reading

Efficient Polygon Fill Algorithm With C Code Sample by Darel Rex Finley explains the algorithm I used for drawing the filled polygons.

256-Color VGA Programming in C by David Brackeen is an old tutorial on VGA graphics programming for DOS. Many things can still be applied.

The DIYConsole series by Davide Pesce. Even though written for Arduino the article series a great job explaining many of the aspects required for writing a graphics library.

Lode’s Fire Effect Tutorial by Lode Vandevenne shows how to create the old school fire effect shown in the header image. Code for ESP32 can also be found in GitHub.

M5Stack on Fire!

Sat, 19 May 2018 00:00:00 +0000

My life has come to a full circle. I remember sitting in my parents basement in the late 90’s programming the fire effect using mixture of Borland Turbo Pascal and Assembler. I had an Intel 80386 which also had the 80387 coprocessor. I remember downloading Lode’s graphics tutorials from FidoNet. It was all magic.

Last week I found myself reading the same tutorials and creating the same fire effect again. Instead of Intel PC, it was written for ESP32 based M5Stack and using C programming language.

Last time I tried to learn C was about eight years ago. After a while I gave up. My first electronics project was an Internet controllable xylophone which recorded a video of the song the user created and sent a download link to email. Like a poor man’s version of the Absolut Machines. Some photos are still in the old Flickr album.

Now is my second time for me to learn C. It seems to go better this time. I do admit I often suffer from the not invented here syndrome. While learning I started two new open source projects: ESP ILI9341 driver and Copepod. Latter is a lightweight hardware agnostics graphics library. Both were used in the fire effect demo.

ILI9341 display driver

For this one I really cannot take the credit. It is mostly taken from SPI Master example found at the ESP-IDF repository. I just extracted the important parts of code and made it more driver like. The driver provides two graphical functions: first one for setting a pixel and second one for blitting an RGB565 bitmap.

#include <driver/spi_master.h>
#include "ili9341.h"

spi_device_handle_t spi;
ili9341_init(&spi);

ili9431_putpixel(spi, x0, y0, color);
ili9431_blit(spi, x0, y0, w, h, &bitmap)

Graphical functions use DMA so it is pretty fast. With 48kHz SPI clock the driver can blit a 16bit 320x240 bitmap approximately 29 times per second. In other words even when using a virtual framebuffer you will get almost 30 fps.

Song is Boink by the excellent Ozzed.

Copepod graphics library

I wanted to call the graphics library Plankton. You know, something small. GitHub search revealed several projects named Plankton so I switched to another small sea critter. Copepod is a lightweight hardware agnostic graphics library. It currently supports only basic geometric primitives, bitmaps, blitting, fixed width fonts and provides an optional framebuffer implementation.

pod_putpixel(x0, y0, color);
pod_line(x0, y0, x1, y1, color);
pod_hline(x0, y0, width, color);
pod_vline(x0, y0, width, color);
pod_rectangle(x0, y0, x1, y1, color);
pod_fillrectangle(x0, y0, x1, y1, color);

For text output the fonts must be of same format as dhepper/font8x8.

pod_putchar(ascii, x0, y0, color, font);
pod_puttext(&str, x0, y0, color, font);

Blit copies a bitmap to the screen. You can also copy a bitmap scaled up or down.

pod_blit(x0, y0, &bitmap);
pod_scale_blit(x0, y0, w, h, &bitmap);

Copepod does not know anything about the underlying hardware. It must be provided with HAL which actually puts the pixels on the screen. Fire demo uses ESP ILI9341 HAL which in turn uses the ESP ILI9341 driver.

Fire fire fire!

I will not go into details how the fire effect works. Read the classic fire tutorial for thorough explanation. Below is an overview of the different tasks used to build the demo. Full source code can be found in GitHub.

If you want to run the code yourself first install ESP-IDF toolchain. Make sure you can compile the example successfully. When that is done compile and install the fire demo with the commands below.

$ git clone git@github.com:tuupola/esp-examples.git
$ cd esp-examples
$ git submodule update --init
$ cd 010-m5stack-fire
$ make menuconfig
$ make flash

For M5stack only needed menuconfig change is the serial flasher port which should be /dev/cu.SLAB_USBtoUART.

Main

The main program starts four FreeRTOS tasks. Note that FreeRTOS is not needed in any way for the effect. However since ESP-IDF already provides it, it is a really convenient way to divide everything into smaller tasks.

xTaskCreatePinnedToCore(framebuffer_task, "Framebuffer", 8192, NULL, 1, NULL, 0);
xTaskCreatePinnedToCore(fps_task, "FPS", 4096, NULL, 2, NULL, 1);
xTaskCreatePinnedToCore(fire_task, "Fire", 8192, NULL, 1, NULL, 1);
xTaskCreatePinnedToCore(switch_task, "Switch", 2048, NULL, 1, NULL, 1);

Note how the framebuffer tast is pinned to the core 0. This frees all the cpu in core 1 to render the effect. Since framebuffer uses DMA also core 0 is mostly idling.

Fire task

This task is responsible of generating the fire effect itself. While the framebuffer has full 320x240 resolution, the fire effect has only 110x80 pixels. I had problems fitting everything in memory. With lower resolution comes speed though. Effect is rendered approximately 45 times per second.

The 110x80 fire effect is rendered into a temporary bitmap. When ready it is scaled up to 320x220 pixels and blitted to the virtual framebuffer. 20 pixels on top of the screen are reserved to the statusbar.

FPS task

FPS task simply displays the top status bar which show the fps rate for both the effect and the framebuffer.

Switch task

The demo has three different effects: burning M5Stack text, sine scroller and full screen fire. This task simply switches between them.

Framebuffer task

Until now everything has been written into a virtual framebuffer. It is a struct which contains a 320x240 RGB565 bitmap. Framebuffer task blits this buffer to the display as fast as it can in an endless loop.

How to Calibrate a Magnetometer?

Fri, 23 Feb 2018 00:00:00 +0000

Magnetometers are used to measure the strength of a magnetic field. They can also be used to determine orientation and to compensate gyro drift. Magnetometer provides the last three degrees of freedom in 9DOF sensors.

There is one problem though, magnetometers are prone to distortion.

Hard iron distortion

The magnetic field used for determining the heading is the earth’s magnetic field. In addition to earth’s, there are additional magnetic fields which cause interference. Interference can be caused by ferromagnetic material or equipment in the magnetometers vicinity. If the magnetic field is permanent it is called “hard iron”.

For example a mobile phone has a speaker. The speaker is permanently attached to the phone. Because of this the location and the orientation of the speakers magnetic field does not change over time. For the magnetometer inside the phone the speaker is considered hard iron.

Good thing about hard iron bias is that it can be easily corrected. Hard iron distortion is always additive to the to the earth’s magnetic field. In other words sensor reading can be corrected ie. unbiased by simply removing the offset. Pseudocode for removing the offset would be something like the following.

offset_x = (max(x) + min(x)) / 2
offset_y = (max(y) + min(y)) / 2
offset_z = (max(z) + min(z)) / 2

corrected_x = sensor_x - offset_x
corrected_y = sensor_y - offset_y
corrected_z = sensor_z - offset_z

Soft iron distortion

Soft iron distortion is the result of material that distorts a magnetic field but does not necessarily generate a magnetic field itself. For example iron (the metal) will generate a distortion but this distorion is dependent upon the orientation of the material relative to the magnetometer.

Unlike hard iron distortion, soft iron distortion cannot be removed by simply removing the constant offset. Correcting soft iron distortion is usually more computation expensive and involves 3x3 transformation matrix.

There is also a computatively cheaper way by using scale biases as explained by Kris Winer. This method should also give reasonably good results. Example pseudocode below includes also the hard iron offset from the previous step.

avg_delta_x = (max(x) - min(x)) / 2
avg_delta_y = (max(y) - min(y)) / 2
avg_delta_z = (max(z) - min(z)) / 2

avg_delta = (avg_delta_x + avg_delta_y + avg_delta_z) / 3

scale_x = avg_delta / avg_delta_x
scale_y = avg_delta / avg_delta_y
scale_z = avg_delta / avg_delta_z

corrected_x = (sensor_x - offset_x) * scale_x
corrected_y = (sensor_y - offset_y) * scale_y
corrected_z = (sensor_z - offset_z) * scale_z

Capture some sensor data

Visualizing the data helps to understand it. It also helps to see the differences after calibrating. For capturing I used the [M5Stack MPU9250 4MB][3] which as the name suggest has a MPU-9250 9DOF sensor inside.

MPU-9250 is a System in Package (SiP) which combines two chips: MPU-6500 which contains 3-axis gyroscope and 3-axis accelerometer and an AK8963 which is a 3-axis digital compass.

Sensor readings are outputted using a MicroPython script to the serial console. Script uses an I2C MPU-9250 driver. After starting the script move the sensor in a big figure eight. Basically the same what you did as a child when playing with toy aeroplane. The sensor should rotate multiple times around the X, Y and Z axles.

import micropython
import utime
from machine import I2C, Pin, Timer
from mpu9250 import MPU9250

micropython.alloc_emergency_exception_buf(100)

i2c = I2C(scl=Pin(22), sda=Pin(21))
sensor = MPU9250(i2c)

def read_sensor(timer):
    value = sensor.magnetic
    print(",".join(map(str, value)))

timer_0 = Timer(0)
timer_0.init(period=500, mode=Timer.PERIODIC, callback=read_sensor)

After approximately 1-2 minutes of waving, copy paste the sensor readings from the console to a csv file. Let’s call it magnetometer.csv. The more data you capture the better.

Seeing is believing

Gnuplot is an excellent cross plaform command line graphing utility. It probably does not appeal to hipster types. There is no unicorn emojis and stuff. It gets the work done though.

If you are on macOS install Gnuplot with Homebrew.

$ brew install gnuplot --with-qt
$ gnuplot

First tell gnuplot input file will be a CSV file.

gnuplot> set datafile separator ","

Scattergraph is good for visualizing magnetometer readings. For three axes we need three separate graphs.

In the plot command using 1:2 states that the values for XY graph are taken from the first and second column. XZ uses data from first and third column, thus using 1:3. YZ is plotted with using 2:3 which are the second and third columns.

gnuplot> plot "magnetometer.csv" using 1:2 title "XY" pointsize 2 pointtype 7, \
              "magnetometer.csv" using 1:3 title "XZ" pointsize 2 pointtype 7, \
              "magnetometer.csv" using 2:3 title "YZ" pointsize 2 pointtype 7

The result should be three similar sized spheres centered around 0,0 coordinates. Here is what I got.

First WTF moment. The graphs do not make any sense. Then I realized M5Stack bottomplate has a magnet which is a schoolbook example of hard iron interference.

New try with the backplate removed.

Much better, but still a lot of distortion. The big offset is still hard iron distortion. Most likely caused by the loudspeaker inside M5Stack. Below is an implementation hard iron offset removal in Python.

#!/usr/local/bin/python3

import csv
import sys

reader = csv.reader(iter(sys.stdin.readline, ""), delimiter=",")
data = list(reader)

x = [float(row[0]) for row in data]
y = [float(row[1]) for row in data]
z = [float(row[2]) for row in data]

offset_x = (max(x) + min(x)) / 2
offset_y = (max(y) + min(y)) / 2
offset_z = (max(z) + min(z)) / 2

for row in data:
    corrected_x = float(row[0]) - offset_x
    corrected_y = float(row[1]) - offset_y
    corrected_z = float(row[2]) - offset_z

    print(",".join(format(value, ".15f") for value in [corrected_x, corrected_y, corrected_z]))

Run the script to generate new csv file with hard iron distortion corrected.

$ ./hardiron.py < magnetometer.csv > hardiron.csv

Plot new graph using the corrected values.

gnuplot> plot "hardiron.csv" using 1:2 title "XY" pointsize 2 pointtype 7, \
              "hardiron.csv" using 1:3 title "XZ" pointsize 2 pointtype 7, \
              "hardiron.csv" using 2:3 title "YZ" pointsize 2 pointtype 7

Result already looks how it should look; three spheres centered around 0,0. There is still some distortion so let’s see what the soft iron removal does.

#!/usr/local/bin/python3

import csv
import sys

reader = csv.reader(iter(sys.stdin.readline, ""), delimiter=",")
data = list(reader)

x = [float(row[0]) for row in data]
y = [float(row[1]) for row in data]
z = [float(row[2]) for row in data]

avg_delta_x = (max(x) - min(x)) / 2
avg_delta_y = (max(y) - min(y)) / 2
avg_delta_z = (max(z) - min(z)) / 2

avg_delta = (avg_delta_x + avg_delta_y + avg_delta_z) / 3

scale_x = avg_delta / avg_delta_x
scale_y = avg_delta / avg_delta_y
scale_z = avg_delta / avg_delta_z

for row in data:
    corrected_x = float(row[0]) * scale_x
    corrected_y = float(row[1]) * scale_x
    corrected_z = float(row[2]) * scale_x

    print(",".join(format(value, ".15f") for value in [corrected_x, corrected_y, corrected_z]))

Run this script using the hard iron corrected values as input.

$ ./softiron.py < hardiron.csv > softiron.csv

Plot the graph again.

gnuplot> plot "softiron.csv" using 1:2 title "XY" pointsize 2 pointtype 7, \
              "softiron.csv" using 1:3 title "XZ" pointsize 2 pointtype 7, \
              "softiron.csv" using 2:3 title "YZ" pointsize 2 pointtype 7

On the first sight difference is not that big. However if you look closer the dots create a bit better formed sphere than before.

Additional reading

Simple and Effective Magnetometer Calibration by Kris Winer which is basis for this blog post. Rest of the wiki is good reading too.

A Way to Calibrate a Magnetometer by Teslabs for more indepth dive on mathematics. Also has example code on how to calibrate using ellipsoid fitting.

Magnetometer Offset Cancellation: Theory and Implementation by William Premerlani describes alternative method for Magnetometer calibrating which can be done on the fly. This is also used in several drone flight controllers.

Calibrating an eCompass in the Presence of Hard- and Soft-Iron Interference ie. the Freescale Semiconductor Application Note AN4246.

Wifi Geolocation With MicroPython

Tue, 06 Feb 2018 00:00:00 +0000

While evaluating M5Stack for a sidehustle project I created a proof of concept which needed to access wifi network, query an API and download an image. Wifi geolocation which displays a static Google map seemed like a perfect fit. Here are some notes about it.

This project is based on the M5Stack kitchen sink. Development was done using Loboris fork of MicroPython. Finished code can be found in GitHub.

Load Settings

The program starts by loading settings from json file. This file contains contains wifi credentials and a Google Maps API key.

{
    "username": "wifiuser",
    "password": "wifipass",
    "api_key": "googlemapsapikey"
}

import ujson as json

with open("/flash/settings.json") as fp:
    settings = json.loads(fp.read())

Scan Nearby Wifi Networks

ESP32 contains two wifi interfaces. STA_IF is the station interface which is used when connecting to a router. STA_AP is the access point interface which is used when other devices connect to the ESP32.

Nearby networks can be scanned with the station interface.

import network

station = network.WLAN(network.STA_IF)
station.active(True)
station.connect(settings["username"], settings["password"])

while not station.isconnected():
    pass

networks = station.scan()

Geolocate Using Wifi Network Data

With the network data in hand we can now find the location of the ESP32 board. For this we need to use a Geolocation API. There are several but the Google Maps Geolocation API is the most well known.

Geolocation API is queried by sending a POST request with a json structure containing the network data. In case wifi location is not found considerIp controls whether to use ip address location as the backup or not.

{
    "considerIp": "false",
    "wifiAccessPoints": [
        {
            "macAddress": "00:25:9c:cf:1c:ac",
            "signalStrength": -43,
            "channel": 10
        },
        {
            "macAddress": "00:25:9c:cf:1c:ad",
            "signalStrength": -55,
            "channel": 8
        }
    ]
}

Python equivalent of above structure can be created as a dict. This can later be serialized to json string with json.dumps().

import ustruct as struct

data = {
    "considerIp": False,
    "wifiAccessPoints": []
}

for wifi in networks:
    entry = {
        "macAddress": "%02x:%02x:%02x:%02x:%02x:%02x" % struct.unpack("BBBBBB", wifi[1]),
        "signalStrength": wifi[3],
        "channel": wifi[2]
    }
    data["wifiAccessPoints"].append(entry)

Making the API requests itself is is done with urequests library. API key was loaded in the beginning from settings.json file. Since we are sending json the correct Content-Type header should also be set.

import ujson as json
import urequests as requests

headers = {"Content-Type": "application/json"}
url = "https://www.googleapis.com/geolocation/v1/geolocate?key=" + settings["api_key"]

response = requests.post(url, headers=headers, data=json.dumps(data))
location = json.loads(response.content)["location"]

Succesful geolocation query returns latitude, longitude and accuracy.

{
    "location": {
        "lat": 51.0,
        "lng": -0.1
    },
    "accuracy": 1200.4
}

The above code ignores accuracy and saves only latitude and longitude to the variable location.

Downloading the Map

The hard part of downloading the map is building the url. Center is the coordinates from previous section. We also put one marker to same location. For size we use 320x240 which is the M5Stack sceen size. Zoom can be anyting between 0 and 20. However something close to 15 is a good start. Format must be jpg-baseline because Loboris display driver does not support progressive jpg files.

query = {
    "center": "%.8f,%.8f" % (location["lat"], location["lng"]),
    "markers": "%.8f,%.8f" % (location["lat"], location["lng"]),
    "size": "320x240",
    "zoom": 15,
    "format": "jpg-baseline"
}

query_string = "&".join("%s=%s" % (key, value) for key, value in query.items())

url = "https://maps.googleapis.com/maps/api/staticmap?" + query_string

With the map url in hand the image can be download again with urequests library.

image = requests.get(url)

Due display drivers lack of support of displayin images from memory buffer, the map image has to be saved to a file. The image variable contains an Response object. Binary image data can be accessed using image.content property.

fp = open("/flash/map.jpg", "wb")
fp.write(image.content)
fp.close()

Display the Map

M5STack has a 320x240 display based on ILI9341 driver. Pin definitions come from the M5Stack kitchen sink helpers. Note that display.TFT() is specific to Loboris fork. It is not available in vanilla MicroPython.

import display
import m5stack

tft = display.TFT()
tft.init(
    tft.ILI9341,
    spihost=tft.HSPI,
    width=320,
    height=240,
    mosi=m5stack.TFT_MOSI_PIN,
    miso=m5stack.TFT_MISO_PIN,
    clk=m5stack.TFT_CLK_PIN,
    cs=m5stack.TFT_CS_PIN,
    dc=m5stack.TFT_DC_PIN,
    rst_pin=m5stack.TFT_RST_PIN,
    backl_pin=m5stack.TFT_LED_PIN,
    backl_on=1,
    speed=2600000,
    invrot=3,
    bgr=True
)

Final step is to display the static map we just downloaded.

tft.image(0, 0, "/flash/map.jpg")

Finished code adds some goodies such as zoom in and out with buttons. Code is also better structured into separate classes.

Install Latest MicroPython to M5Stack

Sat, 27 Jan 2018 00:00:00 +0000

M5Stack is an absolutely beautiful ESP32 based enclosure and development board. It has 320x240 TFT screen, three buttons, sd card slot, Grove I2C connector and can be powered with LiPo battery.

Download the Micropython Firmware

You could download the MicroPython firmware from the downloads page. But why bother when we have the commandline. First find the download link.

$ curl --silent http://micropython.org/download | grep "firmware/esp32" | gawk -F'<a +href="' -v RS='">' 'RT{print $2}'

http://micropython.org/resources/firmware/esp32-20180127-v1.9.3-240-ga275cb0f.bin

$ lynx -listonly -dump https://micropython.org/download | grep "firmware/esp32" | sed 's/.*http/http/'

http://micropython.org/resources/firmware/esp32-20180127-v1.9.3-240-ga275cb0f.bin

When download link is known, download the firmware.

$ wget http://micropython.org/resources/firmware/esp32-20180127-v1.9.3-240-ga275cb0f.bin

--2018-01-27 20:05:26--  http://micropython.org/resources/firmware/esp32-20180127-v1.9.3-240-ga275cb0f.bin
Resolving micropython.org... 176.58.119.26
Connecting to micropython.org|176.58.119.26|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 935888 (914K) [application/octet-stream]
Saving to: ‘esp32-20180127-v1.9.3-240-ga275cb0f.bin’

esp32-20180127-v1.9 100%[===================>] 913.95K   284KB/s    in 3.2s

2018-01-27 20:05:30 (284 KB/s) - ‘esp32-20180127-v1.9.3-240-ga275cb0f.bin’ saved [935888/935888]

Install the Tools

First make sure you have esptool installed. This is the commandline tool which will be used for flashing. Both Python 2 and Python 3 versions should be ok.

$ pip2 install esptool

Collecting esptool
  Downloading esptool-2.2.1.tar.gz (70kB)
    100% |████████████████████████████████| 71kB 94kB/s
Requirement already satisfied: pyserial>=2.5 in /usr/local/lib/python2.7/site-packages (from esptool)
Requirement already satisfied: pyaes in /usr/local/lib/python2.7/site-packages (from esptool)
Requirement already satisfied: ecdsa in /usr/local/lib/python2.7/site-packages (from esptool)
Building wheels for collected packages: esptool
  Running setup.py bdist_wheel for esptool ... done
  Stored in directory: /Users/hello/Library/Caches/pip/wheels/e8/63/0f/01c12901098b6be9d1e7e3e5bc50ef92f1f44a45534095aefa
Successfully built esptool
Installing collected packages: esptool
Successfully installed esptool-2.2.1

Erase the Flash

Connect the M5Stack to your computer with the provided USB cable. Make sure esptool can connect.

$ esptool.py --port /dev/cu.SLAB_USBtoUART --baud 115200 --after no_reset read_mac

esptool.py v2.1
Connecting........_
Detecting chip type... ESP32
Chip is ESP32D0WDQ6 (revision 0)
Uploading stub...
Running stub...
Stub running...
MAC: 30:ae:a4:04:f0:d4
Staying in bootloader.

If esptool connected successfully continue by erasing the flash. Erasing is not absolutely necessary, but usually a good idea to avoid random problems.

$ esptool.py --port /dev/cu.SLAB_USBtoUART  --baud 115200 --after no_reset erase_flash

esptool.py v2.1
Connecting......
Detecting chip type... ESP32
Chip is ESP32D0WDQ6 (revision 0)
Uploading stub...
Running stub...
Stub running...
Erasing flash (this may take a while)...
Chip erase completed successfully in 4.5s
Staying in bootloader.

Flash the new Firmware

The Micropython binary contains bootloader, partition table and the firmare itself concatenated in one file. You should flash this file at offset 0x1000. I use relatively low speed of 115200 for flashing. It takes bit longer but in my experience has less problems.

$ esptool.py --port /dev/cu.SLAB_USBtoUART --baud 115200 write_flash --flash_mode dio --flash_freq 80m --flash_size detect 0x1000 esp32-20180127-v1.9.3-240-ga275cb0f.bin

esptool.py v2.1
Connecting......
Detecting chip type... ESP32
Chip is ESP32D0WDQ6 (revision 0)
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash params set to 0x022f
Compressed 935888 bytes to 587152...
Wrote 935888 bytes (587152 compressed) at 0x00001000 in 51.9 seconds (effective 144.4 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting...

You should now have brand Micropython firmware running on your M5Stack.

Make Sure Everything Works

After flashing connect to the serial REPL to make sure everything is ok. Press enter couple of times to see the prompt.

$ screen /dev/cu.SLAB_USBtoUART 115200
>>>

Picocom is also popular alternative for serial communication. You can install it from Homebrew.

$ brew install picocom
$ picocom -b 115200 /dev/cu.SLAB_USBtoUART
>>>

If you do not get the prompt try removing and reattaching the usb cable. After getting the prompt you can use uos module the make sure you are running the latest version. With gc you can check available memory.

>>>
MicroPython v1.9.3-240-ga275cb0f on 2018-01-27; ESP32 module with ESP32
Type "help()" for more information.
>>> import gc
>>> gc.collect()
>>> gc.mem_free()
90208
>>> import uos
>>> uos.uname()
(sysname='esp32', nodename='esp32', release='1.9.3', version='v1.9.3-240-ga275cb0f on 2018-01-27', machine='ESP32 module with ESP32')
>>>

Update Pycom WiPy Firmware From Commandline

Sun, 24 Sep 2017 00:00:00 +0000

WiPy is a bit pricey but good quality ESP32 development board from Pycom. I got mine from Pimoroni. Pycom does offer their own firmware updater but for my taste it has too much magic going on. I prefer to do things from commandline and I stay in control and to see exactly what update is doing. Quick Googling did not reveal anything so here is my upgrade procedure.

Download the Latest Pycom Firmware

For starters we need to find the latest Pycom firmware. Links can be found from their forums. Or you can just query the filename from their software update service.

$ curl "https://software.pycom.io/findupgrade?key=wipy.wipy%20with%20esp32&redirect=false&type=all"

{
  "key":"wipy.wipy with esp32",
  "file":"https://software.pycom.io/downloads/WiPy-1.7.10.b1.tar.gz",
  "version":"1.8.0.b1",
  "intVersion":71303297,
  "hash":"f6c30bf8b08c50d188f727822be320061ed7847b"
}

At the time of writing the version number and file name do not seem to match. The download itself works though.

$ wget https://software.pycom.io/downloads/WiPy-1.7.10.b1.tar.gz
--2017-09-23 23:24:46--  https://software.pycom.io/downloads/WiPy-1.7.10.b1.tar.gz

Resolving software.pycom.io... 13.58.86.194
Connecting to software.pycom.io|13.58.86.194|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 781586 (763K) [application/octet-stream]
Saving to: 'WiPy-1.7.10.b1.tar.gz'

WiPy-1.7.10.b1.tar. 100%[=====================>] 763.27K   301KB/s   in 2.5s

2017-09-23 23:24:49 (301 KB/s) - 'WiPy-1.7.10.b1.tar.gz' saved [781586/781586]

Or you can just combine above to a single command.

$ wget --content-disposition "https://software.pycom.io/findupgrade?key=wipy.wipy%20with%20esp32&redirect=true&type=all"

--2017-09-24 00:52:27--  https://software.pycom.io/findupgrade?key=wipy.wipy%20with%20esp32&redirect=true&type=all
Resolving software.pycom.io... 13.58.86.194
Connecting to software.pycom.io|13.58.86.194|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://software.pycom.io/downloads/WiPy-1.7.10.b1.tar.gz [following]
--2017-09-24 00:52:27--  https://software.pycom.io/downloads/WiPy-1.7.10.b1.tar.gz
Reusing existing connection to software.pycom.io:443.
HTTP request sent, awaiting response... 200 OK
Length: 781586 (763K) [application/octet-stream]
Saving to: 'WiPy-1.7.10.b1.tar.gz'

WiPy-1.7.10.b1.tar. 100%[=====================>] 763.27K   714KB/s   in 1.1s

2017-09-24 00:52:29 (714 KB/s) - 'WiPy-1.7.10.b1.tar.gz' saved [781586/781586]

After the firmware is downloaded, unpack the tarball to a temporary folder.

$ mkdir temp
$ cd temp
$ tar -xzvf ../WiPy-1.7.10.b1.tar.gz

x bootloader.bin
x partitions.bin
x script
x wipy.bin

You are now almost ready for the flashing.

Install the Tools

First make sure you have esptool installed. This is the commandline tool which will be used for flashing. Both Python 2 and Python 3 versions should be ok.

$ pip2 install esptool

Collecting esptool
Requirement already satisfied: pyaes in /usr/local/lib/python2.7/site-packages (from esptool)
Requirement already satisfied: ecdsa in /usr/local/lib/python2.7/site-packages (from esptool)
Requirement already satisfied: pyserial>=2.5 in /usr/local/lib/python2.7/site-packages (from esptool)
Installing collected packages: esptool
Successfully installed esptool-2.1

Prepare the Board

Before flashing you must put the board to a programming mode. Disconnect USB to make sure the board is not powered. Connect pin G23 to GND. This is easiest to do if you also have the expansion board. The pins are fourth from the top on the left and second from the top on the right side.

Connect the USB cable again. Board should now boot to the programming mode. You can test this with esptool.

$ esptool.py --port /dev/cu.usbserial* --after no_reset read_mac

esptool.py v2.1
Connecting....
Detecting chip type... ESP32
Chip is ESP32D0WDQ6 (revision 0)
Uploading stub...
Running stub...
Stub running...
MAC: 30:ae:a4:00:86:68
Staying in bootloader.

If you instead see something like below it means board is not in programming mode.

$ esptool.py --port /dev/cu.usbserial* --after no_reset read_mac

esptool.py v2.1
Connecting........_____....._____....._____....._____....._____....._____....._____....._____....._____....._____

A fatal error occurred: Failed to connect to Espressif device: Invalid head of packet ('\x08')

If programming mode failed try again. With some board revisions it might help if you hit the reset button once after booting.

Flash the new Firmware

The tarball contains bootloader, partition table and the firmware itself. Usually you do not update the bootloader and partition table. However for completeness sake in this example I update everything.

The ESP32 bootloader lives at flash offset 0x1000. I use relatively low speed of 115200 for flashing. It takes bit longer but in my experience has less problems.

$ esptool.py --port /dev/cu.usbserial* --baud 115200 --after no_reset write_flash --flash_mode dio --flash_freq 80m --flash_size detect 0x1000 bootloader.bin

esptool.py v2.1
Connecting....
Detecting chip type... ESP32
Chip is ESP32D0WDQ6 (revision 0)
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash params set to 0x022f
Compressed 12896 bytes to 8144...
Wrote 12896 bytes (8144 compressed) at 0x00001000 in 0.7 seconds (effective 140.2 kbit/s)...
Hash of data verified.

Leaving...
Staying in bootloader.

After bootloader flash the partition table. It goes to to offset 0x8000. Otherwise command is the same.

$ esptool.py --port /dev/cu.usbserial* --baud 115200 --after no_reset write_flash --flash_mode dio --flash_freq 80m --flash_size detect 0x8000 partitions.bin

esptool.py v2.1
Connecting....
Detecting chip type... ESP32
Chip is ESP32D0WDQ6 (revision 0)
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 3072 bytes to 145...
Wrote 3072 bytes (145 compressed) at 0x00008000 in 0.0 seconds (effective 766.0 kbit/s)...
Hash of data verified.

Leaving...
Staying in bootloader.

Finally flash to firmware itself. Offset is 0x10000. Since the image is quite much larger this command will take a while.

$ esptool.py --port /dev/cu.usbserial* --baud 115200 --after no_reset write_flash --flash_mode dio --flash_freq 80m --flash_size detect 0x10000 wipy.bin

esptool.py v2.1
Connecting......
Detecting chip type... ESP32
Chip is ESP32D0WDQ6 (revision 0)
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 1286416 bytes to 771494...
Wrote 1286416 bytes (771494 compressed) at 0x00010000 in 68.6 seconds (effective 150.0 kbit/s)...
Hash of data verified.

Leaving...
Staying in bootloader.

You should now have brand new firmware running on your WiPy board.

Make Sure Everything Works

After flashing is done disconnect the USB cable and remove the wire between G23 and GND. Connect USB cable again to boot the board and connect to the serial REPL. Press enter couple of times to see the prompt.

$ screen /dev/cu.usbserial* 115200
>>>

If you do not get the prompt try pressing the reset button. After getting the prompt you can use uos module the make sure you are running the latest version.

>>> import uos
>>> uos.uname()
(sysname='WiPy', nodename='WiPy', release='1.8.0.b1', version='v1.8.6-760-g90b72952 on 2017-09-01', machine='WiPy with ESP32')

Branca as an Alternative to JWT?

Sun, 06 Aug 2017 00:00:00 +0000

Branca is a catchy name for IETF XChaCha20-Poly1305 AEAD message with an additional version number and timestamp. It is well suited to be used as an authenticated and encrypted API token. Branca specification does not specify the payload format. Among others you can use for example JWT payloads but still have modern encryption and smaller token size provided by Branca.

Currently there are implemenations for JavaScript, Elixir, Go and PHP and a command line tool for creating and inspecting tokens.

Heads up! JWT itself is the payload part of a larger standard called Javascript Object Signing and Encryption (JOSE). That said the term JWT has become ubiquitous when actually referring JSON Web Signature (JWS) or JSON Web Encryption (JWE). In this article the term JWT refers to JWS.

Branca is based on Fernet specification.

What is Fernet?

Fernet takes an user provided message, a secret key and the current time and generates an Authenticated Encrypted (AE) token. Authenticated encryption specifies a way to secure a message so that a 3rd party cannot fake it, alter it nor read it.

Fernet token is a Base64URL encoded concatenation of the following fields:

Version (1B) | Timestamp (8B) | IV (16B) | Ciphertext (*B) | HMAC (32B)

Where HMAC is 256-bit SHA256 HMAC of the concatenation of the following fields:

Version (1B) | Timestamp (8B) | IV (16B) | Ciphertext (*B)

Fernet seems not to be maintained anymore. There has been no updates for the spec in three years. Original developers are in radio silence.

Introducing Branca

Branca token is a modernized version of Fernet. Branca aims to be secure, easy to implement and have a small token size. Branca token is a Base62 encoded concatenation of the following fields:

Version (1B) | Timestamp (4B) | Nonce (24B) | Ciphertext (*B) | Tag (16B)

Main differences to the Fernet spec are:

Instead of AES 128 CBC and SHA256 HMAC, Branca uses XChaCha20-Poly1305 Authenticated Encryption with Additional Data (AEAD).
Instead of of Base64URL encoding, Branca uses Base62 for the string representation of the token.
Instead of 64-bit unsigned big-endian integer, the timestamp is 32-bit unsigned big-endian integer. This is slightly easier to implement and saves a few bytes, but is still valid until year 2106 when integer oveflow will happen.

Alternative to JWT You Say?

Branca specification defines only the external format of the token. Payload format is intentionally undefined. This means you could still use the JWT payload without the JOSE overhead.

For comparison I will use the typical JWT setup seen in the wild. Payload is unencrypted but both header an payload are authenticated using HMAC SHA-256. In other words a JWS token. Header describes the paylod type and signing algorithm.

{ "typ" : "JWT", "alg" : "HS256" }

Payload is a JSON Web Token. This example payload is taken from RFC7515.

{ "iss" : "joe", "exp" : 1300819380, "http://example.com/is_root" : true }

We can generate the token with a little bit of JavaScript.

const key = "supersecretkeyyoushouldnotcommit";
const jwt = require("jsonwebtoken");

const token = jwt.sign({
    "iss" : "joe",
    "exp" : 1300819380,
    "http://example.com/is_root" : true
}, key, {noTimestamp: true});

const payload = jwt.verify(token, key, {ignoreExpiration: true});

console.log(token);
console.log(payload);

The complete signed and Base64URL encoded token is a 168 character string.

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9eyJpc3MiOiJqb2UiLCJleHAiOjEzMDA4M
TkzODAsImh0dHA6Ly9leGFtcGxlLmNvbS9pc19yb290Ijp0cnVlfQnf_uy1ZrKiO9F4vq
hK7mXrqIbDS799K0DhfDi5g0im8

{ iss: 'joe',
  exp: 1300819380,
  'http://example.com/is_root': true }

Now let’s compare this to a Branca token with exactly the same payload.

const key = "supersecretkeyyoushouldnotcommit";
const branca = require("branca")(key);

const json = JSON.stringify({
    "iss" : "joe",
    "exp" : 1300819380,
    "http://example.com/is_root" : true
});

const token = branca.encode(json);
const payload = branca.decode(token);

console.log(token);
console.log(JSON.parse(payload));

Result is a 148 character token. However unlike the above JWT which is only authenticated Branca token is also encrypted.

4ghanLf33KQKBdEKvL5BpAUWXW6YzPbep54y9211upTQx7tzS2smFO5TWUw7PUdNvPh18
2FUIvQkZfp8kAk3XaE1xVQhhhApErJiIjgw4aC6gyOzmTwFvERIkkw2x2SLjvzHYjTA1f
7ihVL06Xh

{ iss: 'joe',
  exp: 1300819380,
  'http://example.com/is_root': true }

We can optimize this example a bit. Since Branca token has a timestamp in the header we can get rid of the exp claim. The example below uses 130081938 as timestamp just to demonstrate the token size. In reality timestamp should be the time when token was generated. Expiration is checked by the consumer when token is decoded.

const key = "supersecretkeyyoushouldnotcommit";
const branca = require("branca")(key);

const json = JSON.stringify({
    "iss" : "joe",
    "http://example.com/is_root" : true
});

const token = branca.encode(json, 1300819380);
const payload = branca.decode(token, 3600); /* 3600 second ttl */

console.log(token);
console.log(JSON.parse(payload));

Now we are down to 125 characters but we still have all the features of the original JWT.

92FHUGUxd0EHJkHRnHtX1XuTHzJBzD2BHH0LhOWBWloRP5pmZ0TpGNUhBEMmjnm1ZpfB7
4Deba2xGshuewIpvWpGDP5xgOi8OAro7h79Lc10QPa7rtyWuYo76XzT

{ iss: 'joe', 'http://example.com/is_root': true }

We can still cut down the token size by using a binary serializer such as MessagePack or Protocol Buffers. This does require more orchestrating though.

const key = "supersecretkeyyoushouldnotcommit";
const branca = require("branca")(key);
const msgpack = require("msgpack5")();

const packed = msgpack.encode({
    "iss" : "joe",
    "http://example.com/is_root" : true
});

const token = branca.encode(packed);
const binary = branca.decode(token);
const payload = msgpack.decode(Buffer.from(binary));

console.log(token);
console.log(payload);

We are now down to 112 characters. Still authenticated and encrypted.

1UhI0E8r3nNk5KbtxgbD9XwgTEKL6eBVILjfg5TTh71veZwMif4IAQSnvy8LG81CGR0uP
Pg75FNXvP7Lci6TJ3cK8b0ZxrNey4ItLIC9FdOiRRk

{ iss: 'joe', 'http://example.com/is_root': true }

What About Macaroons?

Macaroons, Branca and JOSE are different things. JOSE is a standard encoding for authenticated and optionally encrypted JSON payload. Branca is a standard encoding for authenticated encryption for an arbitrary payload.

Macaroons are more abstract. There are no standard encodings or signing algorithms. Macaroons are about delegating authorization. Holder of a macaroon can pass on a subset of the given authority to a third party.

Additional Reading

Javascript Object Signing and Encryption is a Bad Standard That Everyone Should Avoid by Scott Arciszewski describes the problems in JOSE standard from a cryptographers point of view.

Google’s Macaroons in Five Minutes or Less by Brendan Mc Million is the clearest no nonsense explanation if Macaroons I have seen.

Using JSON Web Tokens as API Keys by Damian Schenkelman. Since the article mostly talks about payload all the examples can also be implemented with a Branca token.

WiFi Trilateration With Three or More Points

Mon, 29 May 2017 00:00:00 +0000

The popularity of WiFi networks has been rising rapidly during the last 15 years. In urban areas hotspots are ubiquous. These hotspots together with trilateration algorithms provide a cheap way to find out yours or someone elses location.

Because of my previous work with Apple location services I wanted to figure out how trilateration works. Being a school dropout I am really really bad in mathematics. However I consider myself good at trial and error so I reckon this should be doable.

Where Three Circles Overlap

Basic idea is that if you know the distance to three other locations it is possible to figure out your current location. This is the geometrical way to solve the problem. To test it out I googled the coordinates three cities and calculated their distance from Tallinn.

To calculate the distances I used R. If you are on macOS you can install it with homebrew.

$ brew install homebrew/science/r

Do not be intimidated by the looks of R. It is made by the academic scholars whose first concern is not usability. R can do everything you ever need to do with math and then some more. It is quite convenient for smaller things too.

Below are the haversine distance calculations.

$ R
> library(geosphere)
> distHaversine(c(24.7535, 59.437), c(24.9354, 60.1695)) # Helsinki
> distHaversine(c(24.7535, 59.437), c(26.7251, 58.3806)) # Tartu
> distHaversine(c(24.7535, 59.437), c(24.4971, 58.3859)) # Pärnu
[1] 82175.02
[1] 163311.4
[1] 117932.2

RFiddle

To test this out I also made a Google Maps page for visualizing the logic. On the map I drew three circles with the distance from Tallinn as radius from each city center.

They seem to intersect over Tallinn as expected.

Distance From RSSI

Lets go back for a while. We were talking about WiFi routers. You do know the coordinates of couple hotspots in the vicinity. What is the distance to them?

Distance to a WiFi hotspot can calculated using the received signal strength indicator (RSSI). If the hotspot is located open air this is fairly accurate. If there are obstacles in the way accuracy is detoriated fast. As you already might have guessed, in urban areas the accuracy is notoriously bad.

But hey, it’s something.

RSSI is expressed as a negative number. The more negative the number, the weaker the signal. The closer to zero the number is the stronger the signal is. Anything down to -75 is considered a good signal.

Below is a one algorithm to calculate distance from RSSI. In the algorithm d is the distance in metres,n is the path-loss exponent and Ptx is the transmitter power in dBm.

Choosing value for n is not exact science and it depends on the enviroment. Typical values are: 2 for free space, 2.7 to 3.5 for urban areas, 3.0 to 5.0 in suburban areas and 1.6 to 1.8 for indoors when there is line of sight to the router.

Default transmit power Ptx for DD-WRT based routers is 70mW or 18.5dBm. I use this value for calculations. Transmit power varies a lot between manufacturers.

RSSI = Ptx - 10 * n * log10(d)
d = 10 ^ ((Ptx - RSSI) / (10 * n))

Putting this to R gives us believeable results.

$ R
10 ^ ((18.5 - -70) / (10 * 3.5))
10 ^ ((18.5 - -70) / (10 * 5.0))
[1] 337.7314
[1] 58.88437

RFiddle

Trilateration With Three Points

Circles on the map proved the intersection is what I am looking for. I had no idea how to calculate this though. All searches led to the Trilateration entry in Wikipedia. It has good step by step instructions.

I do not claim I fully understand how it works, but following the instructions I was able to port the trilateration algorithm to R. I had some problems first but then I figured out coordinates should be converted to ECEH system.

library("pracma")

# Earth radius in metres
EARTH_RADIUS <- 6378137

# Helper function for converting to ECEH coordinates
eceh <- function (s) {
    vx <- EARTH_RADIUS * (cos(deg2rad(s[1])) * cos(deg2rad(s[2])))
    vy <- EARTH_RADIUS * (cos(deg2rad(s[1])) * sin(deg2rad(s[2])))
    vz <- EARTH_RADIUS * (sin(deg2rad(s[1])))
    c(vx, vy, vz)
}

# Helper function for normalizing a vector
normalize <- function (x) { x / sqrt(sum(x^2)) }

# Three spheres
s1 <- c(59.43250050, 24.76253500, 36)
s2 <- c(59.43170784, 24.76271075, 54)
s3 <- c(59.43226950, 24.76160953, 62)
P1 <- eceh(s1)
P2 <- eceh(s2)
P3 <- eceh(s3)

ex <- normalize(P2 - P1);
i <- dot(ex, P3 -P1)
ey <- normalize(P3 - P1 - ex * i);
ez <- cross(ex, ey)
d = dist(rbind(P2, P1))
j = dot(ey, P3 - P1)

x = (s1[3] ^ 2 - s2[3] ^ 2 + d ^ 2) / (2 * d);
y = ((s1[3] ^ 2 - s3[3] ^ 2 + i ^ 2 + j ^ 2) / (2 * j)) - ((i / j) * x);
z = sqrt(s1[3] ^ 2 - x ^ 2 - y ^ 2)

triPt = P1 + ex * x + ey * y + ez * z;

latitude = rad2deg(asin(triPt[3] / EARTH_RADIUS));
longitude = rad2deg(atan2(triPt[2], triPt[1]));

paste(latitude, longitude, sep=",")

[1] "59.4321096315479,24.7626895411386"

See the RFiddle for properly commented version. The code works quite well with perfect or near perfect input. I am not sure what causes the slight error. I guess the ECEH conversion is not accurate since it assumes earth is perfectly round.

When we introduce errors to input data the algorithm gets confused.

Even worse if the circles do not overlap the algorithm does not return result at all. Workaround is quite simple though. Gradually increase the radius of the circles until they overlap and then do the calculation.

This is not a mathematically correct solution. It gives reasonably good results though.

With real life data the results are not that good. A is the trilateration result while B is the real location where the scan was made.

Nonlinear Least Squares

Trilateration using a geometrical model is good when you have near perfect input. Like we learned earlier deriving distance from RSSI is just an educated guess. The geometrical model works with two or three points. What if we have four or five points?

There is a better way called Nonlinear Least Squares Fitting. This is a hard concept to grasp at first. It can be used to solve an optimization problem. In our case the problem is finding the point which represents in the best possible way the three or more points of the WiFi routers.

Nonlinear Least Squares Fitting solves this by trying different coordinates, and then summing the squared error ie. distances from this point to location of each WiFi router. Goal is to find a point where the sum of the squared distances is as small as possible.

Think of Nonlinear Least Squares fitting as mathematical brute forcing.

Trilateration With n Points

Luckily for us R has a Nonlinear Least Squares function. It needs locations parameter which should contain both coordinates and distances. We use distm function from geosphere package for the fitting. Distm calculates the haversine distance between two coordinates.

We also need to provide a starting point. This enables the algorithm to find the results faster. Remember, this is kind of brute forcing. If we aproximately already know what we are looking for brute forcing will be faster.

Good starting point is to use averages of all given coordinates. Here we take the same real life data as above but include more hotspots.

library(geosphere)

locations <- data.frame(
    latitude = c(
        59.42606837, 59.42610146, 59.42654852, 59.42609108,
        59.42603039, 59.42666361
    ),
    longitude = c(
        24.72553151, 24.72552969, 24.72467492, 24.72555759,
        24.72565661, 24.72449149
    ),
    distance = c(8, 8, 9, 9, 9, 14)
)

# Use average as the starting point
fit <- nls(
    distance ~ distm(
        data.frame(longitude, latitude),
        c(fitLongitude, fitLatitude)
    ),
    data = locations,
    start = list(
        fitLongitude=mean(locations$longitude),
        fitLatitude=mean(locations$latitude)
    ),
    control = list(maxiter = 1000, tol = 1e-02)
)

# Result
latitude <- summary(fit)$coefficients[2]
longitude <- summary(fit)$coefficients[1]

paste(latitude, longitude, sep=",")

[1] "59.4262604020993,24.7252212766657"

RFiddle

Results are really good. There is only few meters difference between the calculated point A and the real location B.

Additional Reading

Circles and spheres by Paul Bourke has quite down to earth no-nonsense notes and algorithms about dealing with circles and spheres.

Reverse-Engineering Exact Location Information Through Trilateration by Henrik Mühe has a good explanation about trilateration as an optimization problem.

Positioning and Trilateration by Alan Zucconi has more thorough explanation about everything I wrote above. Python implementation of fitting algorithm is also included.

Implementations

Trilateration for PHP implements both geometric intersection and nonlinear least squares methods. Latter needs an R command line intepreter to be installed.

Trilateration.js is a JavaScript port of the Wikipedia example. Another library called trilat has a nonlinear least squares version. For go there is Trilateration which also is a port of Wikipedia example.

Reverse Engineering Apple Location Services Protocol

Wed, 10 May 2017 00:00:00 +0000

Updates: Added new pdf link to the end (2017-05-11). Header is actually length-prefix framed byte strings (2017-05-13).

While working on Whereami I got interested on how Apple location services actually work. I know it is handled by locationd since Little Snitch keeps blocking it. Usual way of inspecting traffic with proxychains did not work since macOS now has something called System Integrity Protection (SIP).

Alternative way was to setup Charles as MITM proxy for an iOS device. After looking at the traffic which was mostly the device phoning home I got what I needed - a location services request.

Location Services Request

The request itself is just application/x-www-form-urlencode with some binary data.

POST /clls/wloc HTTP/1.1
Host: gs-loc.apple.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 97
Proxy-Connection: keep-alive
Accept: */*
User-Agent: locationd/1756.1.15 CFNetwork/711.5.6 Darwin/14.0.0
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: keep-alive

00000000: 00 01 00 05 65 6e 5f 55 53 00 13 63 6f 6d 2e 61  ....en_US..com.a
00000010: 70 70 6c 65 2e 6c 6f 63 61 74 69 6f 6e 64 00 0c  pple.locationd..
00000020: 38 2e 34 2e 31 2e 31 32 48 33 32 31 00 00 00 01  8.4.1.12H321....
00000030: 00 00 00 2d 12 13 0a 11 62 34 3a 35 64 3a 35 30  ...-....b4:5d:50
00000040: 3a 39 34 3a 33 39 3a 62 33 12 12 0a 10 39 38 3a  :94:39:b3....98:
00000050: 31 3a 61 37 3a 65 36 3a 38 35 3a 37 30 18 00 20  1:a7:e6:85:70..
00000060: 64                                               d

Since data does not have gzip header 0x1f8b my second guess was protocol buffers. After all it is all the rage now and all the cool guys are using it. Let’s try to decode.

$ xxd -r request.hex | protoc --decode_raw
Failed to parse input.

That did not work. Maybe there is something extra in the request. Logic says the mac addresses should be part of the data. Let’s try to decode them. They are the blue part of the hex dump.

00000000: 00 01 00 05 65 6e 5f 55 53 00 13 63 6f 6d 2e 61  ....en_US..com.a
00000010: 70 70 6c 65 2e 6c 6f 63 61 74 69 6f 6e 64 00 0c  pple.locationd..
00000020: 38 2e 34 2e 31 2e 31 32 48 33 32 31 00 00 00 01  8.4.1.12H321....
00000030: 00 00 00 2d 12 13 0a 11 62 34 3a 35 64 3a 35 30  ...-....b4:5d:50
00000040: 3a 39 34 3a 33 39 3a 62 33 12 12 0a 10 39 38 3a  :94:39:b3....98:
00000050: 31 3a 61 37 3a 65 36 3a 38 35 3a 37 30 18 00 20  1:a7:e6:85:70..
00000060: 64                                               d

$ xxd -r request2.hex | protoc --decode_raw
Failed to parse input.

Nope. Top part looks like a header. Let’s try to remove the header instead.

00000000: 00 01 00 05 65 6e 5f 55 53 00 13 63 6f 6d 2e 61  ....en_US..com.a
00000010: 70 70 6c 65 2e 6c 6f 63 61 74 69 6f 6e 64 00 0c  pple.locationd..
00000020: 38 2e 34 2e 31 2e 31 32 48 33 32 31 00 00 00 01  8.4.1.12H321....
00000030: 00 00 00 2d 12 13 0a 11 62 34 3a 35 64 3a 35 30  ...-....b4:5d:50
00000040: 3a 39 34 3a 33 39 3a 62 33 12 12 0a 10 39 38 3a  :94:39:b3....98:
00000050: 31 3a 61 37 3a 65 36 3a 38 35 3a 37 30 18 00 20  1:a7:e6:85:70..
00000060: 64                                               d

$ xxd -r request3.hex | protoc --decode_raw
Failed to parse input.

Still no go.

After trial and erroring for a while I decided to brute force it by removing bytes one by one from the beginning to see if it decodes. Here is a bit polished version of that script.

#!/bin/bash
# Try to decode hidden protocol buffers message from binary

size=$(wc -c < $1)

for ((i=1; i<=$size; i++))
do
    dd if=$1 bs=1 skip=$i | protoc --decode_raw
    if [[ $? == 0 ]]
    then
        printf "\n"
        read -p "Removed $i bytes, continue? [Yy] " -n 1 -r
        printf "\n\n"
        if [[ ! $REPLY =~ ^[Yy]$ ]]
        then
            exit 0
        fi
    fi
done

protomower.sh

$ ./protomower.sh request.bin

Running it first three matches seemed like false positives. There was output but some data was garbled. Fourth one feels legit.

45+0 records in
45+0 records out
45 bytes transferred in 0.000063 secs (714938 bytes/sec)
2 {
  1: "b4:5d:50:94:39:b3"
}
2 {
  1: "98:1:a7:e6:85:70"
}
3: 0
4: 100

Removed 52 bytes, continue? [Yy]

Seems like my original idea was quite close. Yellow part is the removed bytes. Blue part is the successfully decoded protocol buffers message.

00000000: 00 01 00 05 65 6e 5f 55 53 00 13 63 6f 6d 2e 61  ....en_US..com.a
00000010: 70 70 6c 65 2e 6c 6f 63 61 74 69 6f 6e 64 00 0c  pple.locationd..
00000020: 38 2e 34 2e 31 2e 31 32 48 33 32 31 00 00 00 01  8.4.1.12H321....
00000030: 00 00 00 2d 12 13 0a 11 62 34 3a 35 64 3a 35 30  ...-....b4:5d:50
00000040: 3a 39 34 3a 33 39 3a 62 33 12 12 0a 10 39 38 3a  :94:39:b3....98:
00000050: 31 3a 61 37 3a 65 36 3a 38 35 3a 37 30 18 00 20  1:a7:e6:85:70..
00000060: 64                                               d

This means request message has four different type of data. In protocol buffers lingo each data type is called a tag. This message has four tags.

1 is a string which contains a mac address. This is most likely a wifi router mac address.
2 is an embedded message which contains 1 as the value. Think of this as a struct or an object.
3 and 4 are integers. Meaning of them is unknown to me. Maybe age since router was last seen or signal to noise ratio.

To prove the hypothesis let try to make a request with different mac addresses. I used a hex editor to edit the binary request file and did a POST request with curl.

00000000: 00 01 00 05 65 6E 5F 55 53 00 13 63 6F 6D 2E 61  ....en_US..com.a
00000010: 70 70 6c 65 2e 6c 6f 63 61 74 69 6f 6e 64 00 0c  pple.locationd..
00000020: 38 2e 34 2e 31 2e 31 32 48 33 32 31 00 00 00 01  8.4.1.12H321....
00000030: 00 00 00 2d 12 13 0a 11 36 34 3a 64 38 3a 31 34  ...-....64:d8:14
00000040: 3a 37 32 3a 36 30 3a 30 63 12 13 0a 11 31 30 3a  :72:60:0c....10:
00000050: 62 64 3a 31 38 3a 35 66 3a 65 39 3a 38 33 18 00  bd:18:5f:e9:83..
00000060: 20 64                                                d

$ curl https://gs-loc.apple.com/clls/wloc --include --request POST --data-binary @request2.bin

HTTP/1.1 400 Bad Request
Date: Sun, 07 May 2017 06:26:06 GMT
Cneonction: Close
Content-Type: text/plain
X-RID: 62904d6c-fe93-47d5-b579-548f9c83297c
Content-Length: 11

Bad Request

No go. What went wrong?

Looking at the dump you can see message is now one byte longer. So there must be a checksum somewhere. This one is pretty obvious. 0x2d is 45 in decimal and the original message was 45 bytes long. New message is 46 bytes long which would be 0x2e in hex. I would also bet the variable is a 16 bit integer ie. 0x002e.

00000000: 00 01 00 05 65 6E 5F 55 53 00 13 63 6F 6D 2E 61  ....en_US..com.a
00000010: 70 70 6c 65 2e 6c 6f 63 61 74 69 6f 6e 64 00 0c  pple.locationd..
00000020: 38 2e 34 2e 31 2e 31 32 48 33 32 31 00 00 00 01  8.4.1.12H321....
00000030: 00 00 00 2e 12 13 0a 11 36 34 3a 64 38 3a 31 34  ...-....64:d8:14
00000040: 3a 37 32 3a 36 30 3a 30 63 12 13 0a 11 31 30 3a  :72:60:0c....10:
00000050: 62 64 3a 31 38 3a 35 66 3a 65 39 3a 38 33 18 00  bd:18:5f:e9:83..
00000060: 20 64                                             d

$ curl https://gs-loc.apple.com/clls/wloc --include --request POST --data-binary @request3.bin

HTTP/1.1 200 OK
X-RID: bb3cc16a-6680-4019-b5d0-fb52e8c8bd5a
Content-Type: text/plain
Content-Length: 4948

Success. Now we know the format of request.

[header][size][message]

Header itself can be dissected furthrer. I originally though these were just magical ASCII control code. However reader in reddit guided me to correct direction. These seem to be length-prefix framed byte strings. I still think 0x0001 indicates start of header though. It also looks like header is null terminated 0x0000.

NUL SOH      /* 0x0001 start of header */
[length]     /* length of the locale string in bytes */
[locale]     /* en_US */
[length]     /* length of the identifier string in bytes */
[identifier] /* com.apple.locationd */
[length]     /* length of the version string in bytes
[version]    /* 8.4.1.12H321 ie. ios version and build */
NUL NUL      /* 0x0000 end of header */
NUL SOH      /* 0x0001 start of header */
NUL NUL      /* 0x0000 end of header */

I am not sure what the last four bytes mean. Maybe it is a placeholder for second header which is just currently empty.

Location Services Response

The response itself is quite large.

00000000: 00 01 00 00 00 01 00 00 13 4a 12 40 0a 10 36 34  .........J.@..64
00000010: 3a 64 38 3a 31 34 3a 37 32 3a 36 30 3a 63 12 2c  :d8:14:72:60:c.,
00000020: 08 80 98 f7 f8 bc ff ff ff ff 01 10 80 98 f7 f8  ................
00000030: bc ff ff ff ff 01 18 ff ff ff ff ff ff ff ff ff  ................
00000040: 01 28 ff ff ff ff ff ff ff ff ff 01 12 30 0a 11  .(...........0..
00000050: 31 30 3a 62 64 3a 31 38 3a 35 66 3a 65 39 3a 38  10:bd:18:5f:e9:8
00000060: 33 12 18 08 a1 a9 d3 40 10 a0 8c db de 26 18 39  3......@.....&.9
00000070: 20 00 28 11 30 08 58 3c 60 ec 01 a8 01 06 12 2e   .(.0.X<`.......
00000080: 0a 0f 30 3a 31 65 3a 31 33 3a 37 3a 39 30 3a 64  ..0:1e:13:7:90:d
...
000012F0: 01 01 12 2f 0a 10 30 3a 32 61 3a 31 30 3a 65 65  .../..0:2a:10:ee
00001300: 3a 35 30 3a 61 34 12 18 08 c0 a3 d5 40 10 b7 c1  :50:a4......@...
00001310: c8 de 26 18 2b 20 00 28 14 30 0e 58 3e 60 e8 01  ..&.+ .(.0.X>`..
00001320: a8 01 0b 12 2f 0a 10 30 3a 31 31 3a 32 31 3a 63  ..../..0:11:21:c
00001330: 63 3a 35 36 3a 33 32 12 18 08 8d ec c9 40 10 91  c:56:32......@..
00001340: 95 cf de 26 18 61 20 00 28 14 30 0f 58 3f 60 c3  ...&.a .(.0.X?`.
00001350: 1a a8 01 01                                        ....

Lets try our poor mans brute forcing again. It works again. Decoded output is approximately 1400 lines long.

$ ./protomower.sh response.bin

2 {
  1: "64:d8:14:72:60:c"
  2 {
    1: 18446744055709551616
    2: 18446744055709551616
    3: 18446744073709551615
    5: 18446744073709551615
  }
}
2 {
  1: "10:bd:18:5f:e9:83"
  2 {
    1: 135582881
    2: 10399172128
    3: 57
    4: 0
    5: 17
    6: 8
    11: 60
    12: 236
  }
  21: 6
}
...

Removed 10 bytes, continue? [Yy]

First one is bit confusing. 18446744073709551615 is 0xfffffffffffffff ie maximum unsigned 64 bit value. This probably means mac address was not found. I have no idea what to think about 18446744055709551616 ie. 0xfffffffbcf1dcc00

Rest of the results are more clear.

2-1 is the mac address
2-2-1 is the latitude 135582881 * pow(10, -8) = 1.35544532
2-2-2 is the longitude 10399172128 * pow(10, -8) = 103.99172128
2-2-3 looks like the location accuracy
2-21 probably the wifi channel

What puzzled me first is why I get 101 results. Then it occurred me that it is 100 successful results. First two are the mac addresses I sent. Rest of them are mac addresses which are in close vicinity to the ones I submitted.

But Still Why 100 Results?

My guess is Apple offloads the trilateration calculations to client. Instead of doing expensive calculations for everyone just return bunch of access points and their coordinates.

If at least three of those are actually visible to the client core location can use the signal level as distance. When you have three coordinates and their distance to target location you can calculate target location with reasonable accuracy.

Below are the access points location services returned while requesting location in Changi.

Having information of hundred access points around you also reduces the need of contacting the location services server again. As long as core location has coordinates of three visible access points it can calculate the location accurately. This can be done even when offline as long as wifi is turned on.

So What Can I Do with This?

You could write userland core location support for programming language which does not do it natively. Although there are easier ways to achieve the same thing.

More interesting would be to write your own location services server to help with some creative debugging of your location enabled apps.

Additional Reading

Application à l’analyse des données de géolocalisation envoyées par un smartphone by François-Xavier Aguessy and Côme Demoustier. I do not read french but this paper has some .proto file examples and Python code which helped me to get started. Protocol seems to have changed since the paper was published though.

Vulnerability Analysis and Countermeasures for WiFi-based Location Services and Application by Jun Liang (Roy) Feng and Guang Gong is good reading in general to understand how WiFi based positioning works.

Gaussian Processes for Signal Strength-Based Location Estimation by Brian Ferris, Dirk Hahnel and Dieter Fox. I do not understand half of the mathematics. However this paper gives good insight of the problems indoor WiFi positioning has.

Discussion in Reddit and Hacker News.

Different Ways of Accessing an API with JavaScript

Thu, 04 May 2017 00:00:00 +0000

Comparing the good old jQuery with modern alternatives. Fetch is an upcoming native standard. Axios is an elegant promise based HTTP client. In the end everything is wrapped together with Vue.js to create an example online base62 decoder.

Creating a Simple API

For demonstrating purposes lets create a simple JSON API. You can also call it a REST API but that might end with endless discussion on what is REST and what is not. As a basis I used a slimmed down version of Slim 3 API Skeleton.

$ composer create-project --no-interaction --stability=dev \
  tuupola/slim-api-skeleton api.base62.io

After installing I removed features such as authentication which are not needed for this demonstration.

API itself contains two routes. First one for encoding data into base62.

$app->post("/encode", function ($request, $response, $arguments) {
    $body = $request->getParsedBody();
    $encoded = (new Base62)->encode($body["data"]);
    return $response->withStatus(200)
        ->withHeader("Content-Type", "application/json")
        ->write(json_encode(["encoded" => $encoded], JSON_UNESCAPED_SLASHES | JSON_PRETTY_PRINT));
});

Second one for decoding the base62 string back to original.

$app->post("/decode", function ($request, $response, $arguments) {
    $body = $request->getParsedBody();
    $decoded = (new Base62)->decode($body["data"]);
    return $response->withStatus(200)
        ->withHeader("Content-Type", "application/json")
        ->write(json_encode(["decoded" => $decoded], JSON_UNESCAPED_SLASHES | JSON_PRETTY_PRINT));
});

Both expect a JSON object as input with data property containing the payload.

$ curl https://api.base62.io/encode \
    --request POST \
    --header "Content-Type: application/json" \
    --data '{ "data": "Hello world!" }'

{
    "encoded": "T8dgcjRGuYUueWht"
}

$ curl https://api.base62.io/decode \
    --request POST \
    --header "Content-Type: application/json" \
    --data '{ "data": "T8dgcjRGuYUueWht" }'

{
    "decoded": "Hello world!"
}

jQuery

jQuery has been the trusted workhorse for a long time. It still gets the job done and does it very well. However altering the DOM manually will get cumbersome with bigger projects. It is also not considered fashionable anymore.

There is a $.post() shortcut. However it will send the request as application/x-www-form-urlencode and you have to construct the JSON string and set the Content-Type header manually.

<p id="encoded">Encoding...</p>
<p id="decoded">Decoding...</p>
<p id="error"></p>

$.ajax("https://api.base62.io/encode", {
    type: "POST",
    data: JSON.stringify({ data: "Hello world!" }),
    contentType: "application/json",
}).done(function (data) {
    $("#encoded").html(data.encoded);
}).fail(function (xhr, status, error) {
    $("#error").html("Could not reach the API: " + error);
});

$.ajax("https://api.base62.io/decode", {
    type: "POST",
    data: JSON.stringify({ data: "T8dgcjRGuYUueWht" }),
    contentType: "application/json",
}).done(function (data) {
    $("#decoded").html(data.decoded);
}).fail(function (xhr, status, error) {
    $("#error").html("Could not reach the API: " + error);
});

JSFiddle

Fetch

Fetch is a modern equivalent to XMLHttpRequest. It provides much more sane API for making HTTP requests. Fetch is still experimental but will be a future standard. Browser support is decent. Unsurprisinly IE does not support fetch but there is there is a polyfill which fixes the problem.

Usage is quite similar to jQuery. You still have to set the headers and convert the payload to JSON string manually. There are no shortcuts for different request methods. There also are some gotchas with error handling. For example a 404 Not Found response will not cause the promise to be rejected. Instead you have to check the response.ok property and throw an error yourself.

In example below I am still altering the DOM manually. This time only with vanilla JavaScript.

<p id="encoded">Encoding...</p>
<p id="decoded">Decoding...</p>
<p id="error"></p>

fetch("https://api.base62.io/encode", {
    method: "POST",
    headers: {
        "Content-Type": "application/json"
    },
    body: JSON.stringify({ data: "Hello world!" }),
    mode: "cors"
}).then(function(response) {
    if (response.ok) {
        return response.json();
    } else {
        throw new Error("Could not reach the API: " + response.statusText);
    }
}).then(function(data) {
    document.getElementById("encoded").innerHTML = data.encoded;
}).catch(function(error) {
    document.getElementById("error").innerHTML = error.message;
});

fetch("https://api.base62.io/decode", {
    method: "POST",
    headers: {
        "Content-Type": "application/json"
    },
    body: JSON.stringify({ data: "T8dgcjRGuYUueWht" }),
    mode: "cors"
}).then(function(response) {
    if (response.ok) {
        return response.json();
    } else {
        throw new Error("Could not reach the API: " + response.statusText);
    }
}).then(function(data) {
    document.getElementById("decoded").innerHTML = data.decoded;
}).catch(function(error) {
    document.getElementById("error").innerHTML = error.message;
});

JSFiddle

Axios With Vue.js

Axios is a promise based HTTP client for the browser and node.js. Usage is similar to fetch() with the exception that axios provides more developer friendly high level API. Data is converted automatically from and to JSON. Error handling has less surprises. Meaning 404 is considered an error.

Example below uses Vue for modifying the DOM. It uses declarative rendering. When value of app.encoded, app.decoded or app.error changes the HTML will also update automatically. For a small example like this the amount of code looks almost the same. In the long run, however framework like Vue will save you from the spaghetti code which will result from writing the UI code yourself.

<div id="app">
  <p>{{ encoded }}</p>
  <p>{{ decoded }}</p>
  <p>{{ error }}</p>
</div>

var app = new Vue({
    el: "#app",
    data: {
        encoded: "Encoding...",
        decoded: "Decoding...",
        error: null
    },
    created: function () {
        var self = this
        axios
            .post("https://api.base62.io/encode", {
                data: "Hello world!"
            })
            .then(function (response) {
                self.encoded = response.data.encoded
            })
            .catch(function (error) {
                self.error = "Could not reach the API: " + error
            })

        axios
            .post("https://api.base62.io/decode", {
                data: "T8dgcjRGuYUueWht"
            })
            .then(function (response) {
                self.decoded = response.data.decoded
            })
            .catch(function (error) {
                self.error = "Could not reach the API: " + error
            })
    }
});

JSFiddle

Slightly modified version of the above code is also live on base62.io.

Other Contenders

There are other contenders such as SuperAgent and qwest. I tried several but have narroved it down to either axios or fetch. I like the simplicity of axios. It is a perfect match for simplicity of Vue.

HTTP Basic Authentication from Database for Slim

Wed, 05 Nov 2014 00:00:00 +0000

HTTP Basic Authentication middleware comes with simple PDO authenticator. It can be used to authenticate users from database. Authenticator assumes username and hashed password are stored in database. Default name for database table is users. Default column names for username and hash are unsurprisingly user and hash. Column and table names can also be set in options. Hash must be created with password_hash() function. Simplest possible table to store user data looks something like this.

CREATE TABLE users (
    user VARCHAR(32) NOT NULL,
    hash VARCHAR(255) NOT NULL
)

You can then insert an user with following.

$user = "root";
$hash = password_hash("t00r", PASSWORD_DEFAULT);

$status = $pdo->exec(
    "INSERT INTO users (user, hash) VALUES ('{$user}', '{$hash}')"
);

With some users in database you can use them in basic auth.

use \Slim\Middleware\HttpBasicAuthentication\PdoAuthenticator;

$pdo = new \PDO("sqlite:/tmp/users.sqlite");

$app = new \Slim\Slim();

$app->add(new \Slim\Middleware\HttpBasicAuthentication([
    "path" => "/admin",
    "realm" => "Protected",
    "authenticator" => new PdoAuthenticator([
        "pdo" => $pdo
    ])
]));

Different database naming

To override default table and column names you can pass them in options.

$app->add(new \Slim\Middleware\HttpBasicAuthentication([
    "path" => "/admin",
    "realm" => "Protected",
    "authenticator" => new PdoAuthenticator([
        "pdo" => $pdo,
        "table" => "accounts",
        "user" => "username",
        "hash" => "hashed"
    ])
]));

Install

You can install latest version using composer. Source is in GitHub.

$ composer require tuupola/slim-basic-auth

Witchcraft for PHP

Sat, 18 Oct 2014 00:00:00 +0000

There are two kinds of people. Those who like their accessors and mutators to start with get and set. This is also what PHP-FIG seems to suggest.

$unicorn = new Unicorn();
$unicorn->setBirthday("1930-24-12")->setColor("rainbow");
print $unicorn->getAge();

It works well with IDE autocompletion. It is also easy to write API documentation using PHPDoc. Then there are those like me who prefer to leave get and set out.

$unicorn = new Unicorn();
$unicorn->birthday("1930-24-12")->color("rainbow");
print $unicorn->age();

Or sometimes even access them as properties.

$unicorn = new Unicorn();
$unicorn->birthday = "1930-24-12";
$unicorn->color = "rainbow";
print $unicorn->age;

The usual way

You have your usual class with boilerplate accessors and mutators.

class Unicorn
{
    private $color;
    private $birthday;

    public function __construct($color = "white", $birthday = null)
    {
        $this->color = $color;
        $this->birthday = $birthday;
    }

    public function getColor()
    {
        return $this->color;
    }

    public function setColor($color)
    {
        $this->color = $color;
        return $this;
    }

    public function getBirthday()
    {
        return $this->birthday;
    }

    public function setBirthday($birthday)
    {
        $this->birthday = DateTime::createFromFormat("Y-m-d", $birthday);
        return $this;
    }

    public function getAge()
    {
        $now = new DateTime();
        return $this->birthday->diff($now)->format("%y years");
    }
}

It all works really nice with ide autocompletes and everything. Problem is code looks ugly. Yes, it is matter of taste. My taste might be different than yours.

$unicorn = new Unicorn();
$unicorn->setBirthday("1930-24-12")->setColor("rainbow");
print $unicorn->getAge();

Magic methods

Witchcraft to the resque. It implements opionated PHP magic methods as traits. If you add Witchcraft\MagicMethods trait you can use pretty methods.

class Unicorn
{
    use \Witchcraft\MagicMethods;

    private $color;
    private $birthday;

    ...
}

$unicorn = new Unicorn();
$unicorn->birthday("1930-24-12")->color("rainbow");
print $unicorn->age();

HEADS UP! You still must write the boilerplate methods. Witchcraft just enables accessing them as properties.

Magic properties

If you add Witchcraft\MagicProperties trait you can use pretty properties.

class Unicorn
{
    use \Witchcraft\MagicProperties;

    private $color;
    private $owner;

    ...
}

$unicorn = new Unicorn();
$unicorn->birthday = "1930-24-12";
$unicorn->color = "rainbow";
print $unicorn->age;

HEADS UP! You still must write the boilerplate methods. Witchcraft just enables accessing them as properties.

Third party code

You can even use it with third party code. However it will work only if the library has properly implemented mutators and accessors. Let’s take League\Url as an example. I love it. It is just all the get and set which make my eyes bleed.

Example from League website looks like following.

use League\Url\Url;

$url = Url::createFromUrl(
    "http://user:pass@www.example.com:81/path/index.php?query=toto+le+heros#top"
);

$query = $url->getQuery();
$query->modify(array("query" => "lulu l'allumeuse", "foo" => "bar"));
$query["sarah"] = "o connors";

$url->setScheme("ftp");
$url->setFragment(null);
$url->setPort(21);
$url->getPath()->remove("path/index.php");
$url->getPath()->prepend("mongo db");
echo $url . PHP_EOL;

/* ftp://user:pass@www.example.com:21/mongo%20db?query=lulu%20l%27allumeuse&foo=bar&sarah=o%20connors */

To use Witchcraft I extend the original class and add the traits.

use League\Url\Url;

Class MagicUrl extends Url
{
    use \Witchcraft\MagicMethods;
    use \Witchcraft\MagicProperties;
}

With magic in place above example code can become like below.

$url = MagicUrl::createFromUrl(
    "http://user:pass@www.example.com:81/path/index.php?query=toto+le+heros#top"
);

$query = $url->query();
$query->modify(array("query" => "lulu l'allumeuse", "foo" => "bar"));
$query["sarah"] = "o connors";

$url->scheme("ftp");
$url->fragment(null);
$url->port(21);
$url->path->remove("path/index.php");
$url->path->prepend("mongo db");
echo $url . PHP_EOL;

/* ftp://user:pass@www.example.com:21/mongo%20db?query=lulu%20l%27allumeuse&foo=bar&sarah=o%20connors */

WARNING! Use Witchcraft on third party code only if you know what you are doing. It can break things. For example when the library already uses magic methods. Check the source first.

Install

You can install latest version using composer. You can also find the source in GitHub.

$ composer require tuupola/witchcraft

Chained Selects Stable Release

Fri, 10 Oct 2014 00:00:00 +0000

Chained is now stable and in feature freeze. In 2.x version class support will be removed. Next major version will use data attributes will be used instead. After all, it is 2014 already!

Legacy syntax removed

Stable version removes legacy syntax. It is not 100% compatible with 0.9.x branch. You must pass all options to remote version as JavaScript object. Example of all possible options below.

$("#series").remoteChained({
    parents : "#mark",
    url : "/api/series.json",
    depends : "#series",
    loading : "Loading.",
    bootstrap : {
        "" : "--",
        "series-3" : "3 series",
        "series-5" : "5 series",
        "series-6" : "6 series",
        "series-7" : "7 series",
        "selected" : "series-3"
    }
});

Class based version does not have any configuration. You only need to pass parent selector.

$("#engine").chained("#series, #model");

Chaining to non select inputs

You can now also chain to other inputs than selects. Usual case would be hidden or text.

<input id="mark" name="mark" value="audi">
<select id="series" name="series">
    <option value="">--</option>
</select>

$("#series").remoteChained({
    parents : "#mark",
    url : "/api/series.json"
});

Three different JSON formats

Plugin now accepts three different JSON formats. Pass either JavaScript object containing value – text pairs for each option.

{
    "" : "--",
    "series-1" : "1 series",
    "series-3" : "3 series",
    "series-5" : "5 series",
    "series-6" : "6 series",
    "series-7" : "7 series",
    "selected" : "series-6"
}

If you need to be able to sort entries on server side use array of arrays…

[
    [ "", "--" ],
    [ "series-1", "1 series" ],
    [ "series-3", "3 series" ],
    [ "series-5", "5 series" ],
    [ "series-6", "6 series" ],
    [ "series-7", "7 series" ],
    [ "selected", "series-6" ]
]

… or array of objects.

[
    { "" : "--" },
    { "series-1" : "1 series" },
    { "series-3" : "3 series" },
    { "series-5" : "5 series" },
    { "series-6" : "6 series" },
    { "series-7" : "7 series" },
    { "selected" : "series-6" }
]

Install

Download latest minified or remote version minified. You can also install with bower.

$ bower install chained

Flat UI Theme for Developers

Sun, 03 Nov 2013 00:00:00 +0000

Grab a Flat UI theme for TextMate 2 and Sublime Text. It is based on colors of the amazing Flat UI Pro framework made by Designmodo. Theme is for oldschool hackers who prefer their terminal windows to be green or yellow on black. It offers just enough highlighting without looking like a rainbow.

** NOTE! ** Designmodo link above contains my referral code. If you buy something from them I will get a reward and be a happy puppy.

Textmate 2

To install download the zipped bundle. Unzip it and doubleclick on the resulting file. It will ask if you want to install the bundle.

Click yes. Open a document in TextMate 2. Choose Flat UI from the theme menu. Now you are all set.

Sublime Text

To install download the theme file and copy it to Sublime Text Packages folder.

$ cp Flat\ UI.tmTheme ~/Library/Application\ Support/Sublime\ Text\ 2/Packages/

To start using the theme go to settings.

Change the color_scheme setting.

Chained Selects for Zepto

Sun, 27 Oct 2013 00:00:00 +0000

New version of Chained mostly updates the remote version. Plugin now supports both Zepto and jQuery. You can avoid initial AJAX requests by bootstrapping select values. You can configure extra values to be sent to server. Since plugin now accepts more configuration options new prettier syntax was implemented. Lastly support for Bower was added.

Zepto support

Since plugin uses :selected you must also include Zepto selector extension. It is not included by default.

<script src="zepto.js"></script>
<script src="zepto.selector.js"></script>
<script src="jquery.chained.js"></script>

After this you can use the plugin as you normally would.

Bootstrapped selects

There are cases when you know values of the selects on page load. For example when linking directly to a saved order or a specific car model. When you know the values you can bootstrap the remote selects by giving the values in the options. This also avoids making the initial AJAX requests. If you have HTML code like below.

<select id="mark" name="mark">
  <option value="">--</option>
  <option value="bmw" selected>BMW</option>
  <option value="audi">Audi</option>n
</select>
<select id="series" name="series">
  <option value="--">--</option>
</select>

To have BMW Series 3 selected by default you can bootstrap the select values.

$("#series").remoteChained({
    parents : "#mark",
    url : "/api/series.json",
    bootstrap : {
        "--":"--",
        "series-3" : "3 series",
        "series-5" : "5 series",
        "series-6" : "6 series",
        "series-7" : "7 series",
        "selected" : "series-3"
    }
});

PRO TIP! Note also the new prettier syntax in the example above. You can pass all configuration options in one hash.

Send additional values

Sometimes you want to send more values than only the parent selects. Lets say you have select for transmission. Content of transmission select changes when user chooses new engine. However in your database possible different transmissions also depend on what series the car is. You have four different selects.

<select id="mark" name="mark">
  <option value="">--</option>
  <option value="bmw" selected>BMW</option>
  <option value="audi">Audi</option>
</select>

<select id="series" name="series">
  <option value="--">--</option>
</select>

<select id="engine" name="engine">
  <option value="--">--</option>
</select>

<select id="transmission" name="transmission">
  <option value="--">--</option>
</select>

Then you could use the following JavaScript code to send values of both #series and #engine to server.

$("#transmission").remoteChained({
    parents : "#engine",
    url : "/api/transmissions.json",
    depends : "#series"
});

PRO TIP! Parent select values are always sent. You do not have include then in depends setting.

Install

Download latest minified or remote version minified. You can also install with bower.

$ bower install chained

Lazy Load with CSS Backgound Image Support

Tue, 15 Oct 2013 00:00:00 +0000

You can now lazyload CSS background images. Bind the plugin to non image element and it will automatically change the background-image style attribute when element is scrolled into view. You can also use effects.

<div class="lazy" data-original="img/bmw_m1_hood.jpg" style="background-image: url('img/grey.gif'); width: 765px; height: 574px;"></div>

$("div.lazy").lazyload({
    effect : "fadeIn"
});

Optional placeholder image

The placeholder image is now optional. If you omit it plugin will use the default which is data uri grey png. One less http request to make.

<img class="lazy" data-original="img/example.jpg" width="765" height="574">

$("img.lazy").lazyload();

Download

Latest source or minified. Plugin has been tested with Safari 6, Chrome 30, Firefox 24 on OSX and IE8, IE9, IE10, IE11, Chrome 28 on Windows and Mobile Safari on iOS 7.02 and 6.1.3. Full changelog in GitHub.

Lazy Load Bugfix Release

Fri, 03 Aug 2012 00:00:00 +0000

This version of Lazy Load is just a bugfix release. Minor release number bump is there because I pulled a new feature in last minute and I did not want to mess up my git branching. Bugfix usually does not normally warrant for the minor bump.

Bugs fixed

Plugin now works correctly when using many instances and they have different container.

$("#column-1 img").lazyload({ container: $("#column-1") });
$("#column-2 img").lazyload({ container: $("#column-2") });
$("#column-3 img").lazyload({ container: $("#column-3") });

Download

Latest source or minified. Plugin has been tested with Safari 5.1, Safari 6, Chrome 20, Firefox 12 on OSX and Chrome 20, IE 8 and IE 9 on Windows and Safari 5.1 on iOS 5 both iPhone and iPad.

Lazy Load 1.7.0 Released

Sat, 28 Jan 2012 00:00:00 +0000

Previous version of Lazy Load gained traction pretty fast. Good patches were submitted to GitHub. This version of plugin mostly concentrates on speed optimization and event handlers.

New Events

Two new events were added. Handler for appear event is called when image appears to viewport but before it is loaded. Handler for load event is called when image is loaded. Both event handler receive two parameters. First parameter elements_left is numbers of images left to load. Second parameter settings is the settings passed to Lazy Load plugin. Inside both handlers this refers to the image dom element.

$("img.lazy").lazyload({
    appear : function(elements_left, settings) {
        console.log(this, elements_left, settings);
    },
    load : function(elements_left, settings) {
        console.log(this, elements_left, settings);
    }
});

New Parameter

New parameter data_attribute was added. It allows custom naming of original image attribute.

$("img.lazy").lazyload({
data_attribute  : "kitten"
});

<img src="/img/placeholder.gif" data-kitten="/img/real-image.png" width="640" height="480" />

Renamed Parameter

Parameter effectspeed was renamed to effect_speed. Old version will work for couple of versions. This parameter has existed before but it was previously undocumented.

Selectors

Viewport selectors got tuned up. Internally they are used to determine when image appears on screen. Speed up is around 25%. You can compare speed tests of 1.6.0 and 1.7.0. While you’re at there click the Run tests button to help me collect better data. Some selector names were added and changed to match Viewport Selectors plugin.

$("img:in-viewport").something();
$("img:below-the-fold").something();
$("img:above-the-top").something();
$("img:left-of-screen").something();
$("img:right-of-screen").something();

Download

Latest source or minified. Plugin has been tested with Safari 5.1, Firefox 3.6, Firefox 7.0, Firefox 8.0 on OSX and Firefox 3.0, Chrome 14 and IE 8 on Windows and Safari 5.1 on iOS 5 both iPhone and iPad.

How to Use 595 Shift Register?

Mon, 02 Jan 2012 00:00:00 +0000

595 series shift registers come in many flavors. SN74HC595 is the most usual. TPIC6B595 is similar but can be used with more power hungry applications. Pin layouts are different but they all operate in the same way.

Shift register is controlled with three pins. They are usually called DATA, LATCH and CLOCK. Chip manufacturers have different names. See the table below for two examples from Texas Instruments.

	74HC595	TPIC6B595
DATA	SER	SER IN
LATCH	RCLK	RCK
CLOCK	SRCLK	SRCK

CLOCK is an constant high - low signal used to synchronize data transfer. Each time CLOCK goes high two things happen. Current value in shift register gets shifted left by one. Last bit is dropped out. First bit will be set to current value of DATA. To write a byte to shift register this has to happen eight times in a loop.

#define LATCH   B0
#define CLOCK   B1
#define DATA    B2

void shift_out(uint8_t data) {
    for(uint8_t i = 0; i < 8; i++) {
        /* Write bit to data port. */
        if (0 == (data & _BV(7 - i))) {
            digital_write(DATA, LOW);
        } else {
            digital_write(DATA, HIGH);
        }

        /* Pulse clock input to write next bit. */
        digital_write(CLOCK, LOW);
        digital_write(CLOCK, HIGH);
    }
}

After calling shift_out() the shift register internally contains new value. To update output pins you must pull LATCH high. This is sometimes called latching on pulsing the latch. Note that it is not enough to hold LATCH high. Data transfer happens on transition from low to high. This is also called rising edge. Code for a led binary counter from 0 to 65535 would look like the following:

int main(void) {

    for(uint16_t i = 0; i < 0xffff; i++) {
        /* Shift high byte first to shift registers. */
        shift_out(i >> 8);
        shift_out(i & 0xff);

        /* Pulse latch to transfer data from shift registers */
        /* to storage registers. */
        digital_write(LATCH, LOW);
        digital_write(LATCH, HIGH);

        _delay_ms(50);
    }

    return 0;
}

The technique above is called bit banging. Bit banging is a technique for serial communications using software instead of dedicated hardware. Good thing is it is cheap to implement. Bad thing is it wastes processing time. Luckily most AVR chips provide an alternative.

Serial Peripheral Interface Bus

The Serial Peripheral Interface Bus or SPI is a synchronous serial data connection. It provides hardware implementation of the clock pulse and writing data serially. SPI terminology differs from bit banging a bit. You can use SPI with three pins only. If you also want to read value back from the slave device you must use fourth pin. This is what MISO is used for.

Bit Bang	SPI	Description
DATA	MOSI	Master Out Slave In
LATCH	SS	Slave Select
CLOCK	SCLK	Serial Clock
-	MISO	Master In Slave Out

Before you can use SPI it must be configured. Main things to do are setting the device to work as master and the data order. Most of the time this is enough. Configuration is done by setting the appropriate bit in the SPI Control Register SPCR. For all configurable options see the Control and Status Registers table. When using SPI you cannot use any arbitrary pins. Instead read your datasheet to find out which pins your microcontroller uses for SPI. Values below are for ATmega32U4 which is used in both Adafruit ATmega32U4 Breakout Board and Teensy USB Development Board.

#define SS   B0
#define SCLK B1
#define MOSI B2
#define MISO B3

void spi_init(void) {
    pin_mode(SCLK, OUTPUT);
    pin_mode(MOSI, OUTPUT);
    pin_mode(SS, OUTPUT); /* Should be output in Master mode. */

    SPCR &= ~(_BV(DORD)); /* MSB first. */
    SPCR |= _BV(MSTR);    /* Act as master. */
    SPCR |= _BV(SPE);     /* Enable SPI. */
}

After SPI is configured reading and writing to it is easy. When byte is written to SPI Data Register SPDR it will be transmitted to slave. Received bytes (if any) are written to hardware receive buffer. Reading SPDR will return the data in receive buffer.

uint8_t spi_transfer(volatile uint8_t data) {
    SPDR = data;
    loop_until_bit_is_set(SPSR, SPIF);
    return SPDR;
}

Why spi_transfer() and not separate spi_read() and spi_write() functions? Internally both master and slave are 8-bit shift registers (do not confuse this with the TPIC6B595 shift register used in article). One bit is shifted from the master to the slave and from the slave to the master simultaneously in one serial clock cycle. After eight SLCK pulses data has been exchanged between master and slave. Slave never sends data to master by itself. Master always must write something to slave.

Main program is essentially the same as when using shift_out() function. Since we do not read anything back we can just ignore return value of spi_transfer() call.

int main(void) {

    spi_init();

    for(uint16_t i = 0; i < 0xffff; i++) {
        /* Shift high byte first to shift registers. */
        spi_transfer(i >> 8);
        spi_transfer(i & 0xff);

        /* Pulse latch to transfer data from shift registers */
        /* to storage registers. */
        digital_write(SPI_SS, LOW);
        digital_write(SPI_SS, HIGH);
    }

    return 0;
}

Control and Status Registers

SPCR Bit #	Name	Description
bit 7	`SPIE`	SPI Interrupt Enable.
bit 6	`SPE`	SPI Enable. When set hardware SPI operations will be enabled.
bit 5	`DORD`	Data Order. When set LSB will be transmitted first.
bit 4	`MSTR`	Master / Slave Select. When set device will act as SPI master.
bit 3	`CPOL`	Clock Polarity. When set leading edge will be falling and trailing edge rising. Vice versa when unset.
bit 2	`CPHA`	Clock Phase. When set data will be sampled on trailing edge of the clock. Vice versa when unset.
bit 1 bit 0	`SPR1` `SPR0`	SPI Clock Rate Select 1 and 0. Used together with `SPI2X` to set the `SCK` frequency.

SPSR Bit #	Name	Description
bit 7	`SPIF`	SPI Interrupt Flag. Set when a serial transfer is complete. An interrupt is generated if `SPIE` in `SPCR` is set and global interrupts are enabled. Cleared by hardware when interrupt handling vector is executed.
bit 6	`WCOL`	Write Collision Flag. Set if the `SPDR` is written during a data transfer.
bit 5 bit 4 bit 3 bit 2 bit 1		Reserved. Always zero.
bit 0	`SPI2X`	Double SPI Speed. Used together with `SPR1` and `SPR0` to set the `SCK` frequency.

SPI2X	SPR1	SPR0	SCK Frequency
0	0	0	f_osc/4
0	0	1	f_osc/16
0	1	0	f_osc/64
0	1	1	f_osc/128
1	0	0	f_osc/2
1	0	1	f_osc/8
1	1	0	f_osc/32
1	1	1	f_osc/64

Wiring TPIC6B595

Wiring is pretty straight forward but depends on the chip used. TPIC6B595 from Adafruit was used when writing this article.

All shift registers should share CLOCK and LATCH signals. When cascading more than one shift register DATA is redirected to next register by connecting SER OUT to SER IN. Outputs are connected to corresponding leds. To enable outputs of TPIC6B595 you must also tie SRCLR pin to positive voltage and G pin to ground.

In image below CLOCK is blue, LATCH is yellow and DATA is orange.

Simple Serial Communications With AVR libc

Sat, 19 Nov 2011 00:00:00 +0000

I like to use various Arduino boards for AVR development. What I do not like are the Arduino libraries. They are often just wrappers around libc functions or rewrites of functions libc already provides. Serial communications is one good example. Arduino provides you with its own implementation of Serial.print(), Serial.println() and Serial.read() methods. At the same time AVR Libc has proven printf(), puts() and getchar() functions. This article explains easy implementation of libc functions used for serial communications.

If you do not have much experience in programming it is probably better to stick with Arduino libraries. They are good at hiding some of the confusing features of embedded programming. However changes are you grow out of them after few projects. Atmel datasheets are not as confusing as they first appear. You might also want to check the finished code of this article.

Configuring UART

AVR microcontrollers have three control and status registers. Register UCSR0A mostly contains status data. UCSR0B and UCSR0C contain all the configuration settings. See the tables in the end of article for all possible values.

AVR Libc provides helper macros for baud rate calculations. Header file requires F_CPU and BAUD to be defined. After including the header file UBRRL_VALUE, UBRRH_VALUE and USE_2X are defined. First two are used to set UART speed. Last one is used to determine if UART has to be configured to run in double speed mode with given baud rate.

UCSZ20 UCSZ01 and UCSZ00 control the data size. Possible sizes are 5-bit (000), 6-bit (001), 7-bit (010), 8-bit (011) and 9-bit (111). Most common used data size is 8-bit.

With above bits we can set most common configuration: no parity, 8 data bits, 1 stop bit.

#define F_CPU 16000000UL
#define BAUD 9600

#include <util/setbaud.h>

void uart_init(void) {
    UBRR0H = UBRRH_VALUE;
    UBRR0L = UBRRL_VALUE;

#if USE_2X
    UCSR0A |= _BV(U2X0);
#else
    UCSR0A &= ~(_BV(U2X0));
#endif

    UCSR0C = _BV(UCSZ01) | _BV(UCSZ00); /* 8-bit data */
    UCSR0B = _BV(RXEN0) | _BV(TXEN0);   /* Enable RX and TX */
}

Writing and Reading From UART

You can transmit data to UART by writing a byte to USART Data Register UDR0. First you have to make sure UART is ready to transmit new data. You can wait until USART Data Register Empty UDRE flag is set. Alternatively you can wait after each byte to transmission be ready. USART Transmit Complete TXC0 is set when transmission is ready.

void uart_putchar(char c) {
   loop_until_bit_is_set(UCSR0A, UDRE0); /* Wait until data register empty. */
   UDR0 = c;
}

void uart_putchar(char c) {
   UDR0 = c;
   loop_until_bit_is_set(UCSR0A, TXC0); /* Wait until transmission ready. */
}

You can receive data from UART by reading a byte from USART Data Register UDR0. USART Receive Complete RXC0 flag is set if to unread data exists in data register.

char uart_getchar(void) {
    loop_until_bit_is_set(UCSR0A, RXC0); /* Wait until data exists. */
    return UDR0;
}

Redirecting STDIN and STDOUT to UART

FDEV_SETUP_STREAM macro can be used to setup a buffer which is valid for stdio operations. Initialized buffer will be of type FILE. You can define separate buffers for input and output. Alternatively you can define only one buffer which works for both input and output. First and second parameters are names of the functions which will be called when data is either read from or written to the buffer.

FILE uart_output = FDEV_SETUP_STREAM(uart_putchar, NULL, _FDEV_SETUP_WRITE);
FILE uart_input = FDEV_SETUP_STREAM(NULL, uart_getchar, _FDEV_SETUP_READ);

FILE uart_io FDEV_SETUP_STREAM(uart_putchar, uart_getchar, _FDEV_SETUP_RW);

To prepare our uart_putchar and uart_getchar function to be used with streams we have to change the definition a bit. To properly format output we also force adding a carriage return after newline has been sent.

void uart_putchar(char c, FILE *stream) {
    if (c == '\n') {
        uart_putchar('\r', stream);
    }
    loop_until_bit_is_set(UCSR0A, UDRE0);
    UDR0 = c;
}

char uart_getchar(FILE *stream) {
    loop_until_bit_is_set(UCSR0A, RXC0); /* Wait until data exists. */
    return UDR0;
}

Now we can redirect both STDIN and STDOUT to UART. This enables us to use AVR Libc provided functions to read and write to serial port.

int main(void) {

    uart_init();
    stdout = &uart_output;
    stdin  = &uart_input;

    char input;

    while(1) {
        puts("Hello world!");
        input = getchar();
        printf("You wrote %c\n", input);
    }

    return 0;
}

Control and Status Registers

UCSR0A Bit #	Name	Description
bit 7	`RXC0`	USART Receive Complete. Set when data is available and the data register has not be read yet.
bit 6	`TXC0`	USART Transmit Complete. Set when all data has transmitted.
bit 5	`UDRE0`	USART Data Register Empty. Set when the `UDR0` register is empty and new data can be transmitted.
bit 4	`FE0`	Frame Error. Set when next byte in the `UDR0` register has a framing error.
bit 3	`DOR0`	Data OverRun. Set when the `UDR0` was not read before the next frame arrived.
bit 2	`UPE0`	USART Parity Error. Set when next frame in the `UDR0` has a parity error.
bit 1	`U2X0`	USART Double Transmission Speed. When set decreases the bit time by half doubling the speed.
bit 0	`MPCM0`	Multi-processor Communication Mode. When set incoming data is ignored if no addressing information is provided.

UCSR0B Bit #	Name	Description
bit 7	`RXCIE0`	RX Complete Interrupt Enable. Set to allow receive complete interrupts.
bit 6	`TXCIE0`	TX Complete Interrupt Enable. Set to allow transmission complete interrupts.
bit 5	`UDRIE0`	USART Data Register Empty Interrupt Enable. Set to allow data register empty interrupts.
bit 4	`RXEN0`	Receiver Enable. Set to enable receiver.
bit 3	`TXEN0`	Transmitter enable. Set to enable transmitter.
bit 2	`UCSZ20`	USART Character Size 0. Used together with `UCSZ01` and `UCSZ00` to set data frame size. Available sizes are 5-bit (000), 6-bit (001), 7-bit (010), 8-bit (011) and 9-bit (111).
bit 1	`RXB80`	Receive Data Bit 8. When using 8 bit transmission the 8th bit received.
bit 0	`TXB80`	Transmit Data Bit 8. When using 8 bit transmission the 8th bit to be submitted.

UCSR0C Bit #	Name	Description
bit 7 bit 6		USART Mode Select 1 and 0. `UMSEL01` and `UMSEL00` combined select the operating mode. Available modes are asynchronous (00), synchronous (01) and master SPI (11).
bit 5 bit 4	`UPM01` `UPM00`	USART Parity Mode 1 and 0. `UPM01` and `UPM00` select the parity. Available modes are none (00), even (10) and odd (11).
bit 3	`USBS0`	USART Stop Bit Select. Set to select 1 stop bit. Unset to select 2 stop bits.
bit 2 bit 1	`UCSZ01` `UCSZ00`	USART Character Size 1 and 0. Used together with with `UCSZ20` to set data frame size. Available sizes are 5-bit (000), 6-bit (001), 7-bit (010), 8-bit (011) and 9-bit (111).
bit 0	`UCPOL0`	USART Clock Polarity. Set to transmit on falling edge and sample on rising edge. Unset to transmit on rising edge and sample on falling edge.

How Does Led Matrix Work?

Fri, 04 Nov 2011 00:00:00 +0000

Led matrices are fun toys. Who would not love blinkenlights? Electronics is hard. Electronics is much harder than programming. I had hard time trying to understand how do the led matrices work. What is best way to learn something? Build one yourself.

Structure of Led Matrix

In a matrix format LEDs are arranged in rows and columns. You can also think of them as y and x coordinates. Lets assume we have 4x4 matrix. Rows would be marked from A to D and columns from 1 to 4. Now we can address each LED by row and column. Top left led would be (A,1). Bottom down led would be (D,4).

Led matrices come in two flavors. Common-row anode (left) and common-row cathode (right).

Figure above shows the different configurations. The difference between these two configurations is how you lit a led. With common-row anode current sources (positive voltage) are attached to rows A..D and currents sinks (negative voltage, ground) to columns 1..4. With common-row cathode current sinks are attached to rows A..D and currents sources to columns 1..4.

For example. To light bottom down led (D,4) of common cathode matrix you would feed positive voltage to column 4 and connect row D to ground. For sake of clarity I will using common-row cathode in examples for the rest of this article.

Building a LED Matrix

To build a 4x4 common-row cathode matrix you will need 16 LEDs, four resistors, some headers and prototyping board. I started by gluing the leds to prototyping board with epoxy glue. This way it is easier to have LEDs beautifully aligned. When gluing the leds make sure long and short legs are aligned the same way.

When glue is dry it is time to bend and solder. First bend all cathodes to left as close to prototyping board as possible. Solder all cathodes in each row together. When cathodes are ready, bend all anodes. Anodes must not touch cathodes. I used piece of plastic tubing to help bending the anodes to form a bridge above cathodes.

Now solder together all anodes in each row. Solder the headers and connect cathode rows directly to the header.

Anode rows are connected to header with current limiting resistors. Value of the resistor depends on the LED used. Check the LED datasheet for forward voltage and current. LED calculator will help you finding out correct resistor. Matrix is now ready for testing.

Addressing Single LED

Connecting ground to row A and positive voltage to column 1 will light the top right LED (A,1).

Connecting ground to row D and positive voltage to column 4 will light the bottom down LED (D,4).

Intuition would say lighting the both (A,1) and (D,4) at the same time is just connecting all the four wires. This is not the case. There are four LEDs which are lit. This is because current is also flowing through (A,4) and (D,1).

Multiplexing and Persistence of Vision

Multiplexing can be used to display arbitrary patterns with led matrices. Multiplexing is sometimes also called scanning. It scans rows (usually from up to down) and lights needed leds only in one row at time. Something like following:

Start by having everything disconnected.
Connect positive voltage all the needed columns.
Connect row to ground. This lights the needed leds in the row.
Disconnect the row and all columns.
Do the same steps one by one to all rows and then start from the beginning.

Do this slowly and you would see blinking LED rows. Do it really fast and human eye can see the whole pattern. Phenomenon is called persistence of vision.

Draw a Pattern

Lets write some simple code for drawing a pattern on the matrix. Note! Even though I am using Arduino board I do not use Arduino libraries nor IDE for developing. I do however like the Arduino pin numbering scheme. Functions pin_mode() and digital_write() work exactly the same way as their Arduino equivalents.

We start by setting up the pins and default state for them.

uint8_t column_pins[4] = { 2, 3, 4, 5 };
uint8_t row_pins[4]    = { 11, 10, 9, 8 };

static void init(void) {

    /* Turn all columns off by setting then low. */
    for (uint8_t x=0; x<4; x++) {
        pin_mode(column_pins[x], OUTPUT);
        digital_write(column_pins[x], LOW);
    }

    /* Turn all rows off by setting then high. */
    for (uint8_t y=0; y<4; y++) {
        pin_mode(row_pins[y], OUTPUT);
        digital_write(row_pins[y], HIGH);
    }
}

To display a pattern on matrix we use draw() function. Bitmap is passed as two dimensional array. Delay is used only to demonstrate persistence of vision.

uint8_t pattern[4][4]  = {{1,0,0,1}, {0,1,0,0}, {0,0,1,0}, {1,0,0,1}};

void draw(uint8_t buffer[4][4], uint8_t delay) {

    for (uint8_t row=0; row<4; ++row) {
        /* Connect or disconnect columns as needed. */
        for (uint8_t column=0; column<4; ++column) {
            digital_write(column_pins[column], buffer[row][column]);
        }

        /* Turn on whole row. */
        digital_write(row_pins[row], LOW);

        _delay_ms(delay);

        /* Turn off whole row. */
        digital_write(row_pins[row], HIGH);
    }
}

To examine persistence of vision effect we draw the pattern with different delays.

uint8_t main(void) {

    init();

    /* With 100ms delay eye can see updating row by row. */
    for (uint8_t i=0; i<10; i++) {
        draw(pattern, 100);
    }

    /* With 10ms delay pattern appears but flickers. */
    for (uint16_t i=0; i<100; i++) {
        draw(pattern, 10);
    }

    /* Withoud delay solid pattern appears. */
    while (1) {
        draw(pattern, 1);
    }

    return 0;
}

Full code can be found from GitHub. Check the output from video below.

iPhone Controlled HTML5 Logo and Color Cube

Wed, 02 Feb 2011 00:00:00 +0000

Heads up! This article was written in 2011 and it exists mostly for historical purposes.

When Apple released iOS 4.2 end of last year I was on a boat trip to Finland. For me the most interesting features were added to Safari browser. I wanted to learn about WebSockets. New DeviceOrientation API just begged to be abused. I had an idea to control the content of laptop browser by tilting and rotating the phone. I had working but ugly code before ship arrived to Helsinki.

If you have short attention span go straight to the HTML5 logo demo.

Code has been sitting on my hard drive since. I cleaned it up during last couple of days. Also added additional eye candy by using the oh-so-hot-at-the-moment HTML5 logo.

How Does It Look?

How Does It Work?

First open demo page with your browser. Browser must support WebSockets. If you are using Safari which also support CSS 3D transforms check HTML5 Logo page. If you are using Chrome try Colour Cube page. When you open a demo page it generates a random PIN number and connects to a WebSocket server. PIN number is used as part of channel name.

var socket = new WebSocket("ws://ws.appelsiini.net:8080/iphone/" + pin);

Next open the mobile page with you iPhone or iPod. Enter PIN number from previous page and press connect. Your mobile browser now connects to same WebSocket server.

var socket = new WebSocket("ws://ws.appelsiini.net:8080/iphone/" + pin);

If the PIN numbers match your browsers are effectively paired. Mobile browser then uses DeviceOrientation API do determine iPhone orientation. When user tilts or turn the device new orientation is sent via WebSocket to server. Server in turn sends the same data to computer browser which is listening on same channel.

Every time HTML5 Logo page receives WebSocket message it animates the logo using CSS 3D transforms.

socket.onmessage = function(event) {
    var payload = JSON.parse(event.data);
    $("#elevator").css("-webkit-transform", "rotateX(" + payload.data.x * -1 + "deg)");
    $("#aileron").css("-webkit-transform", "rotateZ(" + payload.data.y * 1 + "deg)");
    $("#pitch").css("-webkit-transform", "rotateY(" + payload.data.z * 1 + "deg)");
}

Colour Cube is almost the same. Only difference is It uses Pre3d JavaScript 3d rendering engine to draw and animate the cube.

Note! Spinning HTML5 logo is taken from HTML5 demo by Boaz Sender. Color cube is taken from Pre3d demo gallery by Dean McNamee. I did not originally create them. I just adapted them to connect and animate with iPhones.

WebSocket Server

WebSocket server itself is simple Ruby script based on Event Machine and em-websocket. I could have used Pusher which provides a hosted service. However I wanted to use this as learning experience. Also if this demo goes popular I might hit some message limits of Pusher.

Introduction to Marker Clustering With Google Maps

Tue, 04 Nov 2008 00:00:00 +0000

Heads up! This article was written in 2008. The theory itself is still good but the demos are currently broken.

Static Maps API has URL length limit of around 2048 characters. You can hit this limit quickly when adding lot of markers. You can keep URL short by clustering markers together.

Square Based Clustering

Clustering is usually done by dividing map to squares. Square size depends on map zoom level. Markers inside a square are then grouped into cluster. This technique has some limitations. Look at the following image.

Two markers are close to each other. In fact they are so close they are overlapping. Both markers are also the only marker inside their square. Because markers are in separate square they wont be clustered.

Distance Based Clustering

We can also group markers together based on their distance from each other. We could cluster all markers inside 10 kilometer radius together. There is one problem with this approach. Kilometers (and miles) have different meaning in different zoom levels. In zoomed in map it might mean 100 pixels. In zoomed out maps one kilometer might be only one pixel.

There is only one distance unit which does not have this problem: pixels in current zoom level. One pixel on screen is always one pixel on screen. For example we want to cluster all markers which are 20 pixels from each other. I chose 20 pixels because it happens to be the distance after which markers start to overlap each other.

Now the two markers would be clustered since they are inside 20 pixel radius.

Distance Between Two Coordinates on Earth

Distance between two points on earth can be calculated in several ways. Haversine formula is reasonably accurate and widely used. It assumes earth is spherical (in reality earth is slightly ellipsoid). This causes accuracy to be +-2 km when calculating distances of around 20.000 km. 6371.0 km is used as average radius of earth.

Below is PHP implementation of Haversine formula:

function haversineDistance($lat1, $lon1, $lat2, $lon2) {
    $latd = deg2rad($lat2 - $lat1);
    $lond = deg2rad($lon2 - $lon1);
    $a = sin($latd / 2) * sin($latd / 2) +
            cos(deg2rad($lat1)) * cos(deg2rad($lat2)) *
            sin($lond / 2) * sin($lond / 2);
            $c = 2 * atan2(sqrt($a), sqrt(1 - $a));
    return 6371.0 * $c;
}

But didn’t wee need distance in pixels instead? For that we can use Pythagoras' theorem. Pythagoras’ theorem uses cartesian (pixel) coordinates. Some Mercator magic can be used to convert latitude and longitude to pixel x and y values.

You might wonder where did number 268435456 come from? It is half of the earth circumference in pixels at zoom level 21. You can visualize it by thinking of full map. Full map size is 536870912 x 536870912 pixels. Center of the map in pixel coordinates is 268435456,268435456 which in latitude and longitude would be 0,0.

define('OFFSET', 268435456);
define('RADIUS', 85445659.4471); /* $offset / pi() */

function lonToX($lon) {
    return round(OFFSET + RADIUS * $lon * pi() / 180);
}

function latToY($lat) {
    return round(OFFSET - RADIUS *
                log((1 + sin($lat * pi() / 180)) /
                (1 - sin($lat * pi() / 180))) / 2);
}

function pixelDistance($lat1, $lon1, $lat2, $lon2, $zoom) {
    $x1 = lonToX($lon1);
    $y1 = latToY($lat1);

    $x2 = lonToX($lon2);
    $y2 = latToY($lat2);

    return sqrt(pow(($x1-$x2),2) + pow(($y1-$y2),2)) >> (21 - $zoom);
}

Now we have all needed mathematics in place. What to do with them?

Cluster Markers Together

Let’s write example clusterer function. It takes three parameters:

Array of lat and lon locations.
Distance in pixel inside which markers will be clustered.
Current map zoom level.

Function will return another array where coordinates closer than $distance are clustered together.

function cluster($markers, $distance, $zoom) {
    $clustered = array();
    /* Loop until all markers have been compared. */
    while (count($markers)) {
        $marker  = array_pop($markers);
        $cluster = array();
        /* Compare against all markers which are left. */
        foreach ($markers as $key => $target) {
            $pixels = pixelDistance($marker['lat'], $marker['lon'],
                                    $target['lat'], $target['lon'],
                                    $zoom);
            /* If two markers are closer than given distance remove */
            /* target marker from array and add it to cluster.      */
            if ($distance > $pixels) {
                printf("Distance between %s,%s and %s,%s is %d pixels.\n",
                    $marker['lat'], $marker['lon'],
                    $target['lat'], $target['lon'],
                    $pixels);
                unset($markers[$key]);
                $cluster[] = $target;
            }
        }

        /* If a marker has been added to cluster, add also the one  */
        /* we were comparing to and remove the original from array. */
        if (count($cluster) > 0) {
            $cluster[] = $marker;
            $clustered[] = $cluster;
        } else {
            $clustered[] = $marker;
        }
    }
    return $clustered;
}

We can now test clusterer function with array of coordinates.

$markers   = array();
$markers[] = array('id' => 'marker_1',
                    'lat' => 59.441193, 'lon' => 24.729494);
$markers[] = array('id' => 'marker_2',
                    'lat' => 59.432365, 'lon' => 24.742992);
$markers[] = array('id' => 'marker_3',
                    'lat' => 59.431602, 'lon' => 24.757563);
$markers[] = array('id' => 'marker_4',
                    'lat' => 59.437843, 'lon' => 24.765759);
$markers[] = array('id' => 'marker_5',
                    'lat' => 59.439644, 'lon' => 24.779041);
$markers[] = array('id' => 'marker_6',
                    'lat' => 59.434776, 'lon' => 24.756681);

$clustered = cluster($markers, 20, 11);

print_r($clustered);

If you run the code you can see how marker_3, marker_4 and marker_6 are clustered together. This can better be visualized as map screenshot before and after clustering. Blue marker is a cluster.

Real Life Usage?

Obviously making an array of coordinates into a new array of coordinates is not really usefull. However the first clusterer for Static Maps I committed to GitHub uses previously described technique. Clustering a static map takes only two extra lines of code. First create a cluster. Then add it to the map object. Rest is taken care automatically.

$clusterer = Google_Maps_Clusterer::create('distance');
$map->setClusterer($clusterer);

You can see it in action in capital cities of the world map. City locations are parsed from KML file. Note that in closer zooms locations are slightly off. Coordinates have only two decimals of latitude and longitude.

Currently I have demo code only for Static Maps. Serverside clustering for Google Maps API will follow soon. Thats a promise. Cross my heart.

Zoom and Pan Controls With Static Maps

Fri, 24 Oct 2008 00:00:00 +0000

Heads up! This article was written in 2008 and it exists mostly for historical purposes.

Most of the code from previous Static Maps experiments is now put into one clean package. Previously I showed you how to work with markers and bounds. Now we go forward and add zoom and pan controls. It takes only few lines of code. If you just started reading the series check the theory how it works. As a bonus lets add infowindows / bubbles too.

Note! Image above is just a screenshot. You can test final result in the demo.

Create Map and Some Markers

Start by creating new map object and set the size. We also need to give our API key. Markers are positioned on map using location object. Location can be latitude and longitude represented by Google_Maps_Coordinate object. Location can also be map x and y represented by Google_Maps_Point object.

Because we put markers to map the center is calculated automatically. There is no need to call $map->setCenter(). We can also calculate the closest possible zoom with $map->zoomToFit().

require_once 'Google/Maps.php';

$map = Google_Maps::create('static');
$map->setSize('540x300');
$map->setKey(API_KEY);

$coord_1 = new Google_Maps_Coordinate('58.378700', '26.731110');
$coord_2 = new Google_Maps_Coordinate('58.379646', '26.764090');

$marker_1 = new Google_Maps_Marker($coord_1);
$marker_2 = new Google_Maps_Marker($coord_2);

$map->addMarker($marker_1);
$map->addMarker($marker_2);
$map->zoomToFit();

Add Zoom and Pan Controls

Controls are created using Google_Maps_Control::create() factory method. After creating a control you must attach it to a map. This alone is not enough. When panning and zooming new map center or zoom value is passed in query string. Last line passes the values from URL to the map object.

$zoom = Google_Maps_Control::create('zoom');
$map->addControl($zoom);
$pan = Google_Maps_Control::create('pan');
$map->addControl($pan);

$map->setProperties($_GET);

Note! Image above is just a screenshot. You can test working controls in the demo.

Add Infowindows / Bubbles

Infowindows (or bubbles as they are often referred) are represented by Google_Maps_Infowindow object. You can set the content in constructor or using $bubble->setContent() method. Bubbles have a marker attached to them. Clicking the marker will open the infowindow. As with all other items you must attach them to map.

$bubble_1 = new Google_Maps_Infowindow('Foo Bar');
$bubble_2 = new Google_Maps_Infowindow('Pler pop');

$bubble_1->setMarker($marker_1);
$bubble_2->setMarker($marker_2);

$map->addInfowindow($bubble_1);
$map->addInfowindow($bubble_2);

Note! Image above is just a screenshot. You can test working infobubbles in the demo.

Where’s the Source?

If you want to play around with code you can get from github. Patches, improvements and suggestion are welcome.

$ git clone git://github.com/tuupola/php_google_maps.git
$ wget http://github.com/tuupola/php_google_maps/zipball/master

Simple Static Maps With PHP

Fri, 10 Oct 2008 00:00:00 +0000

Heads up! This article was written in 2008 and it exists mostly for historical purposes.

Lately I have been playing with Google Static Maps API a lot. Writing the same things again and again is tedious job. I decided to put the code together as one clean extendable package. Writing object oriented interface for generating URL is trivial. Real meat is having working zoom and pan controls on static map ~~with just 9 lines of code~~ (demo now includes also clickable markers and infowindows).

Code is still alpha quality. API might change any time. But here is a quick walkthrough of current features. We will build the map you see above step by step.

Create a Map Object

Map object is created using Google_Maps::create('static') factory method. If no markers are set you also need to set the center of the map.

require_once 'Google/Maps.php';

$map = Google_Maps::create('static');

$map->setSize('540x300');
$map->setCenter(new Google_Maps_Coordinate('58.368488', '26.768908'));
$map->setZoom(8);
$map->setKey(API_KEY);

Add some markers

Location on map can be in two ways. Latitude and longitude represented by Google_Maps_Coordinate object. Or pixel x and pixel y location represented by Google_Maps_Point object. You can use both when creating a marker.

$coord_1 = new Google_Maps_Coordinate('58.378700', '26.731110');
$coord_2 = new Google_Maps_Coordinate('58.368488', '26.768908');
$coord_3 = new Google_Maps_Coordinate('58.268488', '26.768908');

$marker_1 = new Google_Maps_Marker($coord_1);
$marker_2 = new Google_Maps_Marker($coord_2);
$marker_3 = new Google_Maps_Marker($coord_3);

$marker_1->setColor('green');
$marker_2->setColor('blue');
$marker_3->setColor('orange');

$map->setMarkers(array($marker_1, $marker_2, $marker_3));

Automatically Calculate Zoom and Center

Google Static Maps API can automatically calculate zoom and center for you. However there is no way to know what zoom level it chose. If you need to know automatically calculated zoom and center use $map->zoomToFit() method.

Here we also add new marker using pixel coordinates. Note that we also clear the center of the map we set in the beginning. This allows map to recenter the map according to markers on the map.

$point_1  = new Google_Maps_Point('308107197', '160958681');
$marker_4 = new Google_Maps_Marker($point_1);
$map->setCenter(false);
$map->addMarker($marker_4);
$map->zoomToFit();

Show Marker Bounds

Sometimes you need to be able to visually see bounding box where all the markers fit in.

$map->showMarkerBounds();

Show Map Bounds at Chosen Zoom Level

You can also calculate and show map bounds at any zoom level. Example below displays map bounds at zoom level 8. Map itself is at zoom level 7.

$map->setZoom(7);
$map_bounds = $map->getBounds(8);
$map->setPath($map_bounds->getPath());

Where’s the Source?

If you want to play around with code you can get from github. Patches, improvements and suggestion are welcome.

$ git clone git://github.com/tuupola/php_google_maps.git
$ wget http://github.com/tuupola/php_google_maps/zipball/master

Infowindows With Google Static Maps

Sat, 13 Sep 2008 00:00:00 +0000

Heads up! This article was written in 2008 and it exists mostly for historical purposes.

Previously I showed you how to make a Google Static Map with clickable markers. Several people emailed me to ask how to show infowindow (or infobubble) when marker is clicked. Technique I explain below still needs JavaScript. It is used to open the infowindow. I use jQuery library in the examples.

Image above is just a screenshot. There is a separate page for working demo. Full source code is also available.

Infowindow HTML

First thing we need is HTML code for the infowindow. I wanted it look exactly the same as in Google Maps. I opened a random map and copied the HTML using FireBug inspect console.

I removed all unneseccary code and moved CSS to separate file. Resulting HTML is quite simple. Now we have an empty infowindow ready for use.

Infowindow Content

In this example we have three green markers. Green markers should open infowindow when clicked. We could write full HTML code for three infowindows. In our case it is unneseccary. We will have separate content div but share rest of infowindow code between markers. This way there will be less HTML code on the page.

<div id="bubble">
  ...
  <div id="content_0" class="bubble content"></div>
  <div id="content_1" class="bubble content"></div>
  <div id="content_2" class="bubble content"></div>
</div>

Later when marker is clicked we should figure out which marker it is and choose one content div to display. Other content divs will be hidden.

Which Marker Was Clicked?

Each content div has unique id. We can inject same id to image map we generate. Lets put the id into name attribute of each link.

<map name="marker_map">
  <area shape="circle" coords="75,103,12" href="#" name="content_0">
  <area shape="circle" coords="122,105,12" href="#" name="content_1">
  <area shape="circle" coords="149,125,12" href="#" name="content_2">
</map>

For this we need to alter imagemap generating code. This code we made in previous part of this tutorial.

$counter = 0;
$imagemap = '<map name="marker_map">';
foreach ($marker_array as $marker) {
    ...
    $marker_y = $center_offset_y + $delta_y - 20;
    $imagemap .= sprintf('<area shape="circle" coords="%d,%d,12" href="#"
                          name="content_%d">',
                          $marker_x, $marker_y, $counter);
    $counter++;
}
$imagemap .= '</map>';

Now we have matching id and name between imagemap and content divs. Last thing to do is open the infowindow.

Opening the Infowindow

When page with static map loads infowindow is hidden by default. We will listen for click events in imagemap areas.

$(function() {
    $('map area').bind('click', function(event) {
        /* Here be the dragons. */
    });
});

When marker is clicked following things need to be done:

Previous infowindow (if any) should be hidden.
Content with same id as markers name should be shown.
Infowindow should be repositioned over clicked marker and then shown.

We take a small shortcut here. We do not calculate infowindow position using markers coordinates. Instead we use coordinates received from click event. This is not exactly the same as marker position but close enough. However you can see how infowindow appears in different places if you move your mouse a bit and click one marker several times.

$(function() {
    $('map area').bind('click', function(event) {

        /* Hide previous infowindow. */
        $('div.bubble.content').hide();

        /* Show content div with same id as markers name. */
        var id = '#' + $(this).attr('name');
        $(id).show();

        /* Calculate infowindow position and show it. */
        var bubble_left = event.pageX - 160;
        var bubble_top  = event.pageY - 200;
        $('#bubble').css({left: bubble_left, top: bubble_top}).show();

        return false;
    });

    $('#bubble img.close').bind('click', function(event) {
        $('#bubble').hide();
    });
});

Notice the last three lines? They close the infobubble when cross icon on top right corner is clicked.

What next?

If you have simple map combining Static Maps API an JavaScript infobubble is a blazing fast technique. What if JavaScript is not available at all? Don’t worry. That we will fix next time. In mean while let’s play around with the working demo.

Clickable Markers With Google Static Maps

Sun, 13 Jul 2008 00:00:00 +0000

Heads up! This article was written in 2008 and it exists mostly for historical purposes.

Static map is one big image. Markers are embedded inside the image. You can not use traditional <a href="#"> tags around separate markers. Binding onclick event to separate marker images wont work either. There are no separate markers. Just one large image.

With imagemaps you can specify arbitary areas inside an image which links to given url. Area can be circle, rectangle or polygon. Simple imagemap could look like following:

<map name="marker_map">
  <area shape="circle" coords="75,103,12" href="#">
  <area shape="circle" coords="122,105,12" href="#">
</map>

With imagemaps we can create clickable markers for Google Static Maps. We need to position an imagemap area over each marker. Problem is how to calculate x and y pixel coordinates for each marker.

For this tutorial lets start with with static map you see above. It is created with following PHP code.

require_once 'Net/URL.php';
require_once 'Google/Maps.php';

$map_width  = 512;
$map_height = 300;
$map_size   = $map_width . 'x' . $map_height;
$zoom       = 13;
$markers    = '58.378700,26.731110,green|58.368488,26.768908,green|
               58.379646,26.764090,green';

$api_key = trim(file_get_contents('api_key.txt'));

/* Build the URL's for Static Maps API. */
$url = new Net_URL('http://maps.google.com/staticmap');
$url->addQueryString('center',  $center);
$url->addQueryString('zoom',    $zoom);
$url->addQueryString('markers', $markers);
$url->addQueryString('size',    $map_size);
$url->addQueryString('key',     $api_key);
$demo_map = $url->getUrl();

Calculating Pixel Coordinates for Markers

For imagemap we need to know where markers are located in static map image. To calculate pixel location of marker we need to know the following things:

Latitude and longitude of the center of the map.
Current zoom level.
Size of the static map in pixels.

Static Maps API can automatically calculate needed zoom level and center position of the map. However there is no way of receiving this information back. Instead we have to calculate center latitude and longitude ourselves.

Calculating Map Center

There are several ways of calculating geographic center of two or more points on earth surface. First is to calculate geographic midpoint in cartesian coordinates. Second is to find center of distance. Center of distance is point which has smallest possible distance from all given points. Both these methods give accurate results but require pretty complex calculations.

Easiest way to find map center is to use average latitude and longitude of all given markers. Result is close enough approximation for our needs.

/* Calculate average lat and lon of markers. */
$marker_array = explode('|', $markers);
foreach ($marker_array as $marker) {
    list($lat, $lon, $col) = explode(',', $marker);
    $lat_sum += $lat;
    $lon_sum += $lon;
}
$lat_avg = $lat_sum / count($marker_array);
$lon_avg = $lon_sum / count($marker_array);

/* Set map to calculated center. */
$url->addQueryString('center',  $center);

We also have to convert center latitude and longitude to pixel coordinates in world map. For this we use previously created Google_Maps PHP class.

/* Calculate center as pixel coordinates in world map. */
$center_x = Google_Maps::LonToX($lon_avg);
$center_y = Google_Maps::LatToY($lat_avg);

While were at it lets also calculate center as pixel coordinates in map image. This is really easy. We only need to divide image width and height by two.

/* Calculate center as pixel coordinates in image. */
$center_offset_x = round(512 / 2);
$center_offset_y = round(300 / 2);

Now we know three things about demo map center. Latitude and longitude coordinates are 58.3756113333,26.7547026667. This equals to 308334961,160637460 in world map pixel coordinates. Which in turn equals 256,150 in image pixels coordinates.

To prove our point let’s add a blue marker in our calculated center of map.

$markers .= sprintf('|%s,%s,blue', $lat_avg, $lon_avg);
$url->addQueryString('markers', $markers);

$map_with_center = $url->getUrl();

We still need to make markers clickable…

Calculating Marker Pixel Positions

For each marker we know their latitude and longitude. To find their location on map image following calculations have to be done.

Convert latitude and longitude to pixel x and y coordinates.
Calculate difference between above and map center x and y coordinates.
Convert above difference to match current zoom level.
Add above difference to center pixel coordinates in image.

PHP code below does all above. Note how shift right operator >> is used to convert the difference between center and marker pixel coordinates for current zoom. Maximum zoom is 21. Variable $zoom is the current zoom level. For zoom level 13 we would be doing delta >> (21-13) which equals delta >> 8 which equals delta / 2^8 which in turn equals delta / 256.

foreach ($marker_array as $marker) {
   list($lat, $lon, $col) = explode(',', $marker);
   $target_y = Google_Maps::LatToY($lat);
   $target_x = Google_Maps::LonToX($lon);
   $delta_x  = ($target_x - $center_x) >> (21 - $zoom);
   $delta_y  = ($target_y - $center_y) >> (21 - $zoom);
   $marker_x = $center_offset_x + $delta_x;
   $marker_y = $center_offset_y + $delta_y;
}

Generate Imagemap for Markers

Now we know marker pixel locations on static map. This enables us to generate imagemap for them. We will create clickable circles with radius of 12 pixels. Lets modify code we just wrote. Note that calculated marker location points to markers foot. We want the round head of marker be clickable. That is done by adjusting y coordinate with -20 pixels.

$imagemap = '<map name="marker_map">';
foreach ($marker_array as $marker) {
  ...
  $marker_y = $center_offset_y + $delta_y - 20;
  $imagemap .= sprintf('<area shape="circle" coords="%d,%d,12" href="#">',
                        $marker_x, $marker_y);
}
$imagemap .= '</map>';

For demo’s sake lets also add some jQuery code to notify us when marker is clicked.

$(function() {
    $('map area').bind('click', function() {
        alert('You clicked on marker.');
        return false;
    });
});

Click on green markers.

Conclusion

It takes a bit work but you can have clickable markers with static map. Yes I used JavaScript to indicate when marker was clicked. But that was not the point. Static maps are usually faster to load than full Google Map JavaScript API. You can also create gracefully degrading Google Map for those users with JavaScript disabled. Ability to click marker also gives you ability to open infobubble without JavaScript. That is left for the next part on series of blog posts about Google Maps.

You can find source code to the working demo from svn.

Google Maps Without JavaScript Part 2

Wed, 04 Jun 2008 00:00:00 +0000

In previous part we made a Google Map with sidebar navigation which works even JavaScript turned off. Marker locations and sidebar were parsed from two KML files. Beginning Google Maps Applications is a great book to learn about KML and Google Maps in general.

In this second part of tutorial I show you how to add zoom and pan controls. Again, they work without JavaScript. Be sure to read tutorial part one first. Also check the live demo.

Before going forward I would like to answer a question which has been asked more than once:

Q: What is the point?
A: Two reasons. Websites should degrade gracefully. Page should still have understandable content even if JavaScript is turned off. Second reason is just because it is cool.

Add controls to static map

First we add arrows and plus and minus images on a separate layer. This layer is positioned over the static map. I use Google provided images. Mapki has list of all available images.

Note! In example code I use only filenames as source attribute. Full address would not fit to screen.

<div id="map" style="width:512px; height:300px">
  <img src="<?php print $current_map ?>" />
  <div id="controls">
    <a href=""><img src="north-mini.png" id="pan_up" /></a>
    <a href=""><img src="west-mini.png" id="pan_left" /></a>
    <a href=""><img src="east-mini.png" id="pan_right" /></a>
    <a href=""><img src="south-mini.png" id="pan_down" /></a>
    <a href=""><img src="zoom-plus-mini.png" id="zoom_in" /></a>
    <a href=""><img src="zoom-minus-mini.png" id="zoom_out" /></a>
  </div>
</div>

Did you notice the $current_map variable? It is Google Static Maps address constructed with PHP. I use Net_URL class to build the address. It makes altering the query string later much easier. Address for current static map is built with following code:

/* Build the URL's for Static Maps API. */
$url = new Net_URL('http://maps.google.com/staticmap');
$url->addQueryString('center',  $_GET['center']);
$url->addQueryString('zoom',    $_GET['zoom']);
$url->addQueryString('markers', $_GET['markers']);
$url->addQueryString('size',   '512x300');
$url->addQueryString('key',     $api_key);

$current_map = $url->getUrl();

Make zoom buttons work

To make zoom buttons work we need to create two URL addresses. First with zoom parameter one smaller than current. Second with zoom level one larger than current. Maximum zoom level is 17. Minimum zoom level is 1.

/* Create query strings for + and - buttons. */
$url->addQueryString('zoom', min(17, $_GET['zoom'] + 1));
$zoom_in = $url->getQueryString();

$url->addQueryString('zoom', max(1, $_GET['zoom'] - 1));
$zoom_out = $url->getQueryString();
</code>

Note that we store only query string to $zoom_in and $zoom_out parameter. Full address is not needed. These query strings are then used as links for zoom in and zoom out controls.

<a href="?<?php print $zoom_in ?>"><img src="zoom-plus-mini.png" id="zoom_in" /></a>
<a href="?<?php print $zoom_out ?>"><img src="zoom-minus-mini.png" id="zoom_out"

Make pan buttons work

Pan buttons are bit more trickier. Map is positioned using center parameter containing latitude and longitude. Lets assume we want to pan map 100 pixels right. For that we need to do following things.

Calculate current center longitude as pixels.
Add 100 pixels to it.
Calculate new pixel value as longitude.
Pass new longitude to Static Maps API in center parameter.

Latitude and longitude to pixels conversions are done using some Mercator projection magic. I do not claim I understand it. Quite frankly, I don’t. Thanks to Bratliff from Google Maps API mailinglist who pointed me to some JavaScript mercator code. I used this as a basis and ported the functions to PHP.

Using the freshly created Google Maps PHP class we can calculate new latitudes and longitudes for each panning direction. Note! In example we move map 60 pixels vertically and 100 pixels horizontally.

list($lat, $lon) = explode(',', $_GET['center']);

/* Calculate new latitudes and longtitudes for each arrow. */
$lat_up    = Google_Maps::adjustLatByPixels($lat, -60,  $_GET['zoom']);
$lat_down  = Google_Maps::adjustLatByPixels($lat,  60,  $_GET['zoom']);
$lon_left  = Google_Maps::adjustLonByPixels($lon, -100, $_GET['zoom']);
$lon_right = Google_Maps::adjustLonByPixels($lon,  100, $_GET['zoom']);

Next we put new center parameter into query string. Each direction is stored into separate parameter.

/* Make query strings out of them. */
$url->addQueryString('center', "$lat_up,$lon");
$pan_up    = $url->getQueryString();

$url->addQueryString('center', "$lat_down,$lon");
$pan_down  = $url->getQueryString();

$url->addQueryString('center', "$lat,$lon_left");
$pan_left  = $url->getQueryString();

$url->addQueryString('center', "$lat,$lon_right");
$pan_right = $url->getQueryString();

These query strings are later used as links for panning controls.

<a href="?<?php print $pan_up ?>"><img src="north-mini.png" id="pan_up" /></a>
<a href="?<?php print $pan_left ?>"><img src="west-mini.png" id="pan_left" /></a>
<a href="?<?php print $pan_right ?>"><img src="east-mini.png" id="pan_right" /></a>
<a href="?<?php print $pan_down ?>"><img src="south-mini.png" id="pan_down" /></a>

Now we have non JavaScript map with working sidebar, zoom and panning. Info windows (bubbles) are still missing. Lets leave that for part 3. You can find source code to the working demo from svn.

Google Maps Without JavaScript

Sat, 26 Apr 2008 00:00:00 +0000

We recently did a small Google Maps application for ERGO insurance. It consisted of submitting all their offices to Google Maps. KML export of office data was used to create map at ERGO autoabi campaign site. First version used all JavaScript approach to create the sidebar and map on the page. I was not happy with it. Map page was empty for browsers with JavaScript disabled. Page also took too long to render. Two problems I could not ignore.

Second version was mixture of PHP, JavaScript and new static maps API. Map now works without JavaScript. It renders much faster too. Check the working demo to see yourself.

Importing KML

Google offers GGeoXML library for KML parsing. That I ditched in the beginning. It was impossible to create sidebar navigation with it. First I ended up using EGeoXML. It got the job done but as I said earlier was too slow.

To speed things up I parsed KML files and outputted sidebar HTML with PHP. Simplified HTML and PHP below. Do not mind about # hrefs. They will be replaced with something meaningfull later.

<h3><a id="cityname" class="city" href="#">ERGO cityname</a></h3>
<ul>
    <li><a  class="office" href="#" id="marker-1">Office 1</a></li>
    <li><a  class="office" href="#" id="marker-2">Office 2</a></li>
    <li><a  class="office" href="#" id="marker-3">Office 3</a></li>
</ul>

$kml = array("tartu.kml", "parnu.kml");

/* Used in id's so we can later bind click to correct marker. */
$counter  = 0;
foreach($kml as $file) {
    $city = str_replace('.kml', '', $file);
    $xml  = $xml = simplexml_load_file($file, null, LIBXML_NOCDATA);

    /* Print cityname as headline. */
    printf('<h3><a id="%s" class="city" href="#">%s</a></h3>',
            $city, $xml->Document->name);
    foreach ($xml->Document->Placemark as $placemark) {
        $coordinates = $placemark->Point->coordinates;
        list($longtitude, $latitude, $discard) = explode(',', $coordinates, 3);

        /* Save parsed KML as simpler array. We output this as JSON later. */
        /* Save also id so we can match clicked link ad marker.            */
        $markers[] = array("latitude"    => $latitude,
                           "longtitude"  => $longtitude,
                           "name"        => (string)$placemark->name,
                           "description" => (string)$placemark->description,
                           "id"          => "marker-$counter");

        /* Print officename as link to single office. */
        printf('<li><a class="office" href="#" id="marker-%d">%s</a></li>',
                $latitude, $longtitude, $counter, $placemark->name);
        print "\n";
        $counter++;
    }
    print "</ul>\n";
}

Putting Markers on Map With JavaScript

I gave the PHP parsed KML array back to JavaScript using JSON. Array is then looped passing data to addMarker() method. This method creates new marker, puts it on map and binds event to open new infowindow when marker is clicked. Simplified code below:

$(function() {

    /* Output parsed KML files as JSON */
    var markers     = <?php print json_encode($markers) ?>;
    var marker_hash = {};

    if (GBrowserIsCompatible()) {
        /* Init and center at Tartu. */
        var map = new GMap2(document.getElementById("map"));
        map.addControl(new GSmallMapControl());
        map.setCenter(new GLatLng(58.38133351447725, 24.516592025756836), 12);

        for (var i=0; i<markers.length; i++) {
            var current = markers[i];
            var marker  = addMarker(current);
            marker_hash[current.id] = {marker : marker};
        }
    }

    function addMarker(current) {
      var marker  = new GMarker(new GLatLng(current.latitude,
                                            current.longtitude));
      map.addOverlay(marker);
      GEvent.addListener(marker, 'click', function() {
          var html = '<h3>' + current.name + '</h3><p>' +
                     current.description + '</p>';
          marker.openInfoWindowHtml(html);
      });
      return marker;
    }

});

Make It Work With JavaScript Challenged Browsers

Inside the map <div> there is an <img> tag which points to Google Static maps api. You can pass needed parameters in URL and Google generates map as static image. For example:

http://maps.google.com/staticmap?&zoom=12&size=688x300
&center=58.378700,26.731110&key=GOOGLE_API_KEY

Generates the following map:

What we do here is really simple. In sidebar links we pass needed parameters in querystring. PHP then writes these parameters into <img> tag pointing to static maps api. Instead of center we pass markers parameter. It contains coordinates to one or more markers. Map api then centers map automatically. This comes especially handy when there is multiple markers.

For example link:

<a href="?markers=58.384933,24.499811,red|58.384731,24.507816,red|">

becomes the following img tag:

<img src="http://maps.google.com/staticmap?size=688x300
&markers=58.384933,24.499811,red|58.384731,24.507816,red|
&key=GOOGLE_API_KEY" />

which looks like this:

What we need to do is to update the code which generates sidebar html. I cut out lots of code for sake of clarity here. Important line is one with printf().

foreach($kml as $file) {
    ...
    foreach ($xml->Document->Placemark as $placemark) {
        ...
        printf('<li><a  class="office" href="?zoom=14&markers=%s,%s,red"
        id="marker-%d">%s</a></li>', $latitude, $longtitude, $counter,
        $placemark->name);
        ...
    }
    ...
}

Each link now shows you map with single marker centered on map. Creating link for each city is requires one extra step. Each city contains multiple markers. Before creating the sidebar we loop through KML once to to build a querystring for marker parameter.

/* Prebuild querystring for all markers in a city. This is needed  */
/* to make a link which contains all markers in query string.      */
foreach($kml as $file) {
    $city = str_replace('.kml', '', $file);
    $temp = 'markers=';
    $xml = $xml = simplexml_load_file($file, null, LIBXML_NOCDATA);
    foreach ($xml->Document->Placemark as $placemark) {
        $coordinates = $placemark->Point->coordinates;
        list($longtitude, $latitude, $discard) = explode(',', $coordinates, 3);
        $temp .= sprintf('%s,%s,red|', $latitude, $longtitude);
    }
    $markers_string[$city] = $temp;
}

We also need to make small change code building the sidebar.

foreach($kml as $file) {
    ...
    /* Use prebuild querystring in city links. */
    printf('<h3><a id="%s" class="city" href="?%s">%s</a></h3>',
    $city, $markers_string[$city], $xml->Document->name);
    ...
}

Now we have working map with sidebar. Biggest difference is missing infowindows (bubbles). They would be perfectly doable but I left them out for a reason now.

You can find source code to the working demo from svn. There was couple of cosmetic things I did not describe here so check the source. ERGO Autoabi minisite still contains both original and improved versions of the map.

Static maps api is really great. I have only one complaint. Maximum size of 512x512 is a bit too limiting to my taste.

UPDATE: Google recently increased the maximum size of static map to 640x640. Thank you!

Base32 encoder and decoder for arbitrary data

Mon, 01 Jan 0001 00:00:00 +0000

This library implements Base32 encoding. In addition to integers it can encode and decode any arbitrary data.

Install

Install with composer.

$ composer require tuupola/base32

Usage

This package has both pure PHP and GMP based encoders. By default encoder and decoder will use GMP functions if the extension is installed. If GMP is not available pure PHP encoder will be used instead.

$base32 = new Tuupola\Base32;

$encoded = $base32->encode(random_bytes(128));
$decoded = $base32->decode($encoded);

If you are encoding to and from integer use the implicit decodeInteger() and encodeInteger() methods.

$integer = $base32->encodeInteger(987654321); /* 5N42FR== */
print $base32->decodeInteger("5N42FR=="); /* 987654321 */

Also note that encoding a string and an integer will yield different results.

$integer = $base32->encodeInteger(987654321); /* 5N42FR== */
$string = $base32->encode("987654321"); /* FHE4DONRVGQZTEMI= */

Encoding Modes

RCF4684

RCF4684 is the default when encoder is created without passing any parameters. Explicit parameters shown below.

use Tuupola\Base32;

$base32 = new Base32([
    "characters" => Base32::RFC4648,
    "padding" => "="
]);

print $base32->encode("Hello world!"); /* JBSWY3DPEB3W64TMMQQQ==== */

RCF4684 HEX

RCF4684 base32hex encoding is identical to previous, except for the character set. This encoding is lexically sortable.

$base32hex = new Base32([
    "characters" => Base32::HEX,
    "padding" => "="
]);

print $base32->encode("Hello world!"); /* 91IMOR3F41RMUSJCCGGG==== */

GMP

GMP encoding is identical to previous. Example below is shown with padding disabled.

$gmp = new Base32([
    "characters" => Base32::GMP,
    "padding" => false
]);

print $gmp->encode("Hello world!"); /* 91IMOR3F41RMUSJCCGGG */

Crockford’s Base32

When decoding, upper and lower case letters are accepted, and i and l will be treated as 1 and o will be treated as 0. When encoding, only upper case letters are used. Hyphens are ignored during decoding.

$crockford = new Base32([
    "characters" => Base32::CROCKFORD,
    "padding" => false,
    "crocford" => true,
]);

print $crockford->encode("Hello world!"); /* 91JPRV3F41VPYWKCCGGG */
print $crockford->decode("91JPRV3F41VPYWKCCGGG"); /* Hello world! */
print $crockford->decode("91jprv3f41vpywkccggg"); /* Hello world! */
print $crockford->decode("9ljprv3f4lvpywkccggg"); /* Hello world! */

Z-base-32 [WIP]

http://philzimmermann.com/docs/human-oriented-base-32-encoding.txt

Character Sets

By default Base32 uses RFC4648 character set. Shortcut is provided for other commonly used character sets. You can also use any custom character set of 32 unique characters.

use Tuupola\Base32;

print Base32::CROCKFORD; /* 0123456789ABCDEFGHJKMNPQRSTVWXYZ */
print Base32::RFC4648; /* ABCDEFGHIJKLMNOPQRSTUVWXYZ234567 */
print Base32::ZBASE32; /* ybndrfg8ejkmcpqxot1uwisza345h769 */
print Base32::GMP; /* 0123456789ABCDEFGHIJKLMNOPQRSTUV */
print Base32::HEX; /* 0123456789ABCDEFGHIJKLMNOPQRSTUV */

$default = new Base32(["characters" => Base32::RFC4648]);
$crockford = new Base32(["characters" => Base32::CROCKFORD]);
print $default->encode("Hello world!"); /* JBSWY3DPEB3W64TMMQQQ==== */
print $inverted->encode("Hello world!"); /* 91JPRV3F41VPYWKCCGGG==== */

Speed

Install GMP if you can. It is much faster pure PHP encoder. Below benchmarks are for encoding random_bytes(128) data. BCMatch encoder is also included but it is mostly just a curiosity. It is too slow to be usable.

$ phpbench run benchmarks/ --report=default

+-----------------------+-----------------+----------------+
| subject               | mean            | diff           |
+-----------------------+-----------------+----------------+
| benchGmpEncoder       | 73,099.415ops/s | 0.00%          |
| benchGmpEncoderCustom | 61,349.693ops/s | +19.15%        |
| benchPhpEncoder       | 25.192ops/s     | +290,072.37%   |
| benchBcmathEncoder    | 7.264ops/s      | +1,006,253.07% |
+-----------------------+-----------------+----------------+

Static Proxy

If you prefer to use static syntax use the provided static proxy.

use Tuupola\Base32Proxy as Base32;

$encoded = Base32::encode(random_bytes(128));
$decoded = Base32::decode($encoded);

Testing

You can run tests either manually or automatically on every code change. Automatic tests require entr to work.

$ make test

$ brew install entr
$ make watch

Contributing

Please see CONTRIBUTING for details.

Security

If you discover any security related issues, please email tuupola@appelsiini.net instead of using the issue tracker.

License

The MIT License (MIT). Please see License File for more information.

Chained selects

Mon, 01 Jan 0001 00:00:00 +0000

Chained is simple plugin for chained selects. It works with both jQuery and Zepto. You can choose from two different versions. Use jquery.chained.js if you do not want to make external queries for setting content of child selects. This version uses data attirbutes to decide the content.

For more complex scenarios maintaining data attributes will get cumbersome. Also if you want to make queries against database use jquery.chained.remote.js instead. This version makes an external AJAX query and uses the returned JSON to build child selects.

Install

You can install with yarn, npm or bower.

$ yarn add jquery-chained
$ npm install jquery-chained
$ bower install chained

Simple usage

Child selects are chained to parent select. All selects must have an id attribute. Child select options must have class names which match option values of parent select. When user selects something in parent select the options in child select are updated. Options which have matching classname with parents currently selected option will stay visible. Others are hidden.

First you must include jQuery or Zepto and Chained in your code:

<script src="jquery.js"></script>
<script src="jquery.chained.js"></script>

If you are using Zepto you must also include the optional selector module.

<script src="zepto.js"></script>
<script src="zepto-selector.js"></script>
<script src="jquery.chained.js"></script>

Then lets assume you have the following HTML code:

<select id="mark" name="mark">
    <option value="">--</option>
    <option value="bmw">BMW</option>
    <option value="audi">Audi</option>
</select>
<select id="series" name="series">
    <option value="">--</option>
    <option value="series-3" data-chained="bmw">3 series</option>
    <option value="series-5" data-chained="bmw">5 series</option>
    <option value="series-6" data-chained="bmw">6 series</option>
    <option value="a3" data-chained="audi">A3</option>
    <option value="a4" data-chained="audi">A4</option>
    <option value="a5" data-chained="audi">A5</option>
</select>

You can now chain the series to mark. There are two different ways to do it. Choose yourself if you prefer more english like or shorter version. I prefer the shorter version.

$("#series").chained("#mark"); /* or $("#series").chainedTo("#mark");

Chaining to multiple parents

One child can have two parents. Available options in child which is chained to multiple parents depend on one or both of the parents selected values. To make child select depend on values of both parents use data attribute like first+second.

Here is code for fourth select. Note how diesel engine is available only for BMW 3 and 5 series Sedans. This is achieved by using classnames series-3+sedan and series-5+sedan.

<select id="engine" name="engine">
    <option value="">--</option>
    <option value="25-petrol" data-chained="series-3 a3 a4">2.5 petrol</option>
    <option value="30-petrol" data-chained="series-3 series-5 series-6 a3 a4 a5">3.0 petrol</option>
    <option value="30-diesel" data-chained="series-3+sedan series-5+sedan a5">3.0 diesel</option>
</select>

$("#series").chained("#mark");
$("#model").chained("#series");
$("#engine").chained("#series, #model");

Usage with AJAX

Using Remote Version Using the remote version is similar to what has been explained above. First include jQuery or Zepto and remote version of Chained:

<script src="jquery.js"></script>
<script src="jquery.chained.remote.js"></script>

In HTML you only need to provide option tags for the first select. Contents of other selects will be built from JSON returned by AJAX request. AJAX request is done when value of parent select changes.

<select id="mark" name="mark">
    <option value="">--</option>
    <option value="bmw">BMW</option>
    <option value="audi">Audi</option>
</select>
<select id="series" name="series">
    <option value="">--</option>
</select>
<select id="model" name="model">
    <option value="">--</option>
</select>
<select id="engine" name="engine">
    <option value="">--</option>
</select>

In code you must use remoteChained() method. Second parameter is URL where the AJAX request is sent.

$("#series").remoteChained({
    parents : "#mark",
    url : "/api/series"
});

$("#model").remoteChained({
    parents : "#series",
    url : "/api/models"
});

$("#engine").remoteChained({
    parents : "#series, #model",
    url : "/api/engines"
});

When change event is triggered on parent select a GET request is sent to the given URL. This request includes the name and value of the parent in the query string. For example when users selects BMW in the first select the following request is made:

GET http://example.com/api/series?mark=bmw

JSON data format

By default chained can handle two different formats of JSON response. Object containing key + values pairs is easy to generate. However properties of an object in JavaScript do not have an order. Depending on browser select options might appear on different order.

{
    "" : "--",
    "series-1" : "1 series",
    "series-3" : "3 series",
    "series-5" : "5 series",
    "series-6" : "6 series",
    "series-7" : "7 series",
    "selected" : "series-6"
}

If want to sort the entries on serverside to specific order use array of objects instead.

[
    { "" : "--" },
    { "series-1" : "1 series" },
    { "series-3" : "3 series" },
    { "series-5" : "5 series" },
    { "series-6" : "6 series" },
    { "series-7" : "7 series" },
    { "selected" : "series-6" }
]

If you are accessing third party data source and do not have control over data structure you can use data function. It should mutate and return the json data in one of the above formats. Example below shows how you could mutate namespaced data to a format which plugin uderstands.

{
    "data": [
        { "" : "--" },
        { "series-1" : "1 series" },
        { "series-3" : "3 series" },
        { "series-5" : "5 series" },
        { "series-6" : "6 series" },
        { "series-7" : "7 series" },
        { "selected" : "series-6" }
    ]
}

$("#series").remoteChained({
    parents : "#mark",
    url : "/api/series",
    data: function (json) {
        return json.data;
    }
});

License

All code licensed under the MIT License.

Changelog

See releases.

K-Sortable Globally Unique IDs for PHP

Mon, 01 Jan 0001 00:00:00 +0000

This library implements the K-Sortable Globally Unique IDs from Segment. See also the article called A Brief History of the UUID.

KSUID is for K-Sortable Unique IDentifier. It’s a way to generate globally unique IDs similar to RFC 4122 UUIDs, but contain a time component so they can be “roughly” sorted by time of creation. The remainder of the KSUID is randomly generated bytes.

Install

Install with composer.

$ composer require tuupola/ksuid

This branch requires PHP 7.1 or up. The older 1.x branch supports also PHP 5.6 and 7.0.

$ composer require "tuupola/ksuid:^1.0"

Usage

Included Base62 implementation has both PHP and GMP based encoders. By default encoder and decoder will use GMP functions if the extension is installed. If GMP is not available pure PHP encoder will be used instead.

Note! Throughout the code the term timestamp refers to KSUID timestamp. The term unixtime refers to the traditional Unix time. KSUID timestamp and Unix time have different Epoch.

use Tuupola\Ksuid;

$ksuid = new Ksuid;

print $ksuid; /* p6UEyCc8D8ecLijAI5zVwOTP3D0 */

print $ksuid->timestamp(); /* 94985761 */
print $ksuid->unixtime(); /* 1494985761 */
print bin2hex($ksuid->payload()); /* d7b6fe8cd7cff211704d8e7b9421210b */

$datetime = (new \DateTimeImmutable)
    ->setTimestamp($ksuid->unixtime())
    ->setTimeZone(new \DateTimeZone("UTC"));

print $datetime->format("Y-m-d H:i:s"); /* 2017-05-17 01:49:21 */

If you prefer static syntax you can use one of the provided factories.

use Tuupola\KsuidFactory as Ksuid;

$ksuid = Ksuid::create();

$ksuid = Ksuid::fromString("0o5Fs0EELR0fUjHjbCnEtdUwQe3");

$binary = hex2bin("05a95e21d7b6fe8cd7cff211704d8e7b9421210b");
$ksuid = Ksuid::fromBytes($binary);

$ksuid = Ksuid::fromTimestamp(94985761);

$ksuid = Ksuid::fromUnixtime(1494985761);

$timestamp = 94985761;
$payload = hex2bin("d7b6fe8cd7cff211704d8e7b9421210b");
$ksuid = Ksuid::fromTimestampAndPayload($timestamp, $payload);

Testing

You can run tests either manually or automatically on every code change. Automatic tests require entr to work.

$ make test

$ brew install entr
$ make watch

Contributing

Please see CONTRIBUTING for details.

Security

If you discover any security related issues, please email tuupola@appelsiini.net instead of using the issue tracker.

License

The MIT License (MIT). Please see License File for more information.

Lazy Load Remastered

Mon, 01 Jan 0001 00:00:00 +0000

Lazy Load delays loading of images in long web pages. Images outside of viewport will not be loaded before user scrolls to them. This is opposite of image preloading.

This is a modern vanilla JavaScript version of the original Lazy Load plugin. It uses Intersection Observer API to observe when the image enters the browsers viewport. Original code was inspired by YUI ImageLoader utility by Matt Mlinac. New version loans heavily from a blog post by Dean Hume.

Basic Usage

By default Lazy Load assumes the URL of the original high resolution image can be found in data-src attribute. You can also include an optional low resolution placeholder in the src attribute.

<script src="https://cdn.jsdelivr.net/npm/lazyload@2.0.0-rc.2/lazyload.js"></script>

<img class="lazyload" data-src="img/example.jpg" width="765" height="574" />
<img class="lazyload" src="img/example-thumb.jpg" data-src="img/example.jpg" width="765" height="574" />

With the HTML in place you can then initialize the plugin using the factory method. If you do not pass any settings or image elements it will lazyload all images with class lazyload.

lazyload();

If you prefer you can explicitly pass the image elements to the factory. Use this for example if you use different class name.

let images = document.querySelectorAll(".branwdo");
lazyload(images);

If you prefer you can also use the plain old constructor.

let images = document.querySelectorAll(".branwdo");
new LazyLoad(images);

The core IntersectionObserver can be configured by passing an additional argument

new LazyLoad(images, {
     root: null,
     rootMargin: "0px",
     threshold: 0
});

Additional API

To use the additional API you need to assign the plugin instance to a variable.

let lazy = lazyload();

To force loading of all images use loadimages().

lazy->loadImages();

To destroy the plugin and stop lazyloading use destroy().

lazy->destroy();

Note that destroy() does not load the out of viewport images. If you also want to load the images use loadAndDestroy().

lazy->loadAndDestroy();

Additional API is not avalaible with the jQuery wrapper.

jQuery Wrapper

If you use jQuery there is a wrapper which you can use with the familiar old syntax. Note that to provide BC it uses data-original by default. This should be a drop in replacement for the previous version of the plugin.

<img class="lazyload" data-original="img/example.jpg" width="765" height="574">
<img class="lazyload" src="img/example-thumb.jpg" data-original="img/example.jpg" width="765" height="574">

$("img.lazyload").lazyload();

Cookbook

Blur Up Images

Low resolution placeholder ie. the “blur up” technique. You can see this in action in this blog entry. Scroll down to see blur up images.

<img class="lazyload"
     src="thumbnail.jpg"
     data-src="original.jpg"
     width="1024" height="768" />

<div class="lazyload"
     style="background-image: url('thumbnail.jpg')"
     data-src="original.jpg" />

Responsive Images

Lazyloaded Responsive images are supported via data-srcset. If browser does not support srcset and there is no polyfill the image from data-src will be shown.

<img class="lazyload"
     src="thumbnail.jpg"
     data-src="large.jpg"
     data-srcset="small.jpg 480w, medium.jpg 640w, large.jpg 1024w"
     width="1024" height="768" />

<img class="lazyload"
     src="thumbnail.jpg"
     data-src="normal.jpg"
     data-srcset="normal.jpg 1x, retina.jpg 2x"
     width="1024" height="768" />

Inlined Placeholder Image

To reduce the amount of request you can use data uri images as the placeholder.

<img src="data:image/gif;base64,R0lGODdhAQABAPAAAMPDwwAAACwAAAAAAQABAAACAkQBADs="
     data-src="original.jpg"
     width="1024" height="768" />

Install

This is still work in progress. You can install beta version with yarn or npm.

$ yarn add lazyload
$ npm install lazyload

License

All code licensed under the MIT License. All images licensed under Creative Commons Attribution 3.0 Unported License. In other words you are basically free to do whatever you want. Just don’t remove my name from the source.