What is HTTPS?

One of the significant 'trends' in technology in 2018 (if one can really call it a trend and not a necessity) is the wide adoption of HTTPS as a communications protocol. In this post, I'll attempt to touch on the fundamental components of HTTPS, including authentication, encryption, certificate authorities and public and private keys.

As an aside: later, I'll work up to a fully automated, self renewing CA compliant HTTPS implementation using standard off the shelf java technologies, however this article won't go there.

Background

OK, so HTTPS is a communications protocol. What is a communications protocol and why use one?

The classic examples of a communications protocols are traffic lights. There are different variants of the traffic light protocol in different countries, but basically the traffic light protocol consists of verbs, GO, SLOW and STOP, generally specified by the colours, GREEN, YELLOW and RED. Without the traffic light communications protocol, we wouldn't all know how to drive together on the road!

People tell me pictures paint 1000 words, so below is a picture of a car stopped at a traffic light, just to show that this isn't some fancy whiz bang concept here and I'm talking about real cars receiving messages from real traffic lights using the STOP, GO, SLOW verbs expressed by the colors of the lights.

The internet uses a different communications protocol called HTTP. Like the traffic light protocol, it also uses verbs, but uses six verbs instead of three. These six verbs are named GET, POST, PUT, OPTIONS, HEAD and DELETE, with each having their own specific meaning (these verbs may be the subject of a separate post).

The HTTP communications protocol is what allows the internet to scale, and something like 99.99% of websites on the internet use this protocol for all of their communications. Websites that only show 'outbound' information use the GET verb to fetch their information from a server so they can display it to the user on screen. Websites that additionally collect 'inbound' information from their end users may use the rest.*

* Awful tech pun fully intended (see REST).

No security is built into HTTP by itself, so any additional person or device intercepting information sent using the HTTP protocol (such as a router for example) can read all of that information. This includes passwords, logins and bank account details. Whatever information is displayed or collected using HTTP protocol is visible to anyone or anything that intercepts it, and may as well be public information.

But what about the S?

Yes, the S. The S in HTTPS has been around a long time, since 1994 and the Netscape Navigator days. Initially, it was primarily used to protect sensitive information during financial transactions. Recently though, people have gotten real concerned with privacy. Particularly, they don't like to share their personal data with others, especially as now-a-days personal data can involve payment platform credentials, or information that can lead to payment platform credentials that can be used to steal money. Detailed personal information that is collected (legitimately or illegitimately) can also be sold in dodgy places online. Data hacks are bad and common, with some estimates that personal data breaches and cyber crime cost more than 400 billion US dollars annually. The S in HTTPS alleviates these issues by securing HTTP communications.

What does HTTPS look like? 

How end users have interacted with HTTPS over the years has varied, and many end users may not have even known when they were using HTTPS as it has generally worked  'under the hood' of the web browser.

However, some users may be familiar with URLs in their address bar that start with https:// and in addition, more recently, a small padlock to the left of a URL.

The padlock shows that api.quotecrunchers.com uses a secure https connection.

This contrasts with sites that do not employ HTTPS at all, which increasingly are highlighted by web browsers in an attempt by browser vendors to shame websites into becoming more secure.

Google's Chrome web browser added this address bar warning to http sites that collect user information in July 2018

What does 'secure' mean exactly? 

The S in HTTPS achieves two things from a security perspective.

1) It guarantees authenticity in terms of the identity of the website providing or collecting information. (known as Authentication in the business). It does this by using digital signatures.

2) It encrypts data sent between the client and the sever so that it cannot be read by anyone in between (in what is called a 'man in the middle' attack).

HTTPS Benefit 1: Digital signatures to guarantee authenticity, wait / what? 

Guaranteeing authenticity via digital signatures seems counter intuitive but it is possible. One common attribute of digital information is that it can be copied, so an obvious question is, why can't the crooks at https://www.evilhackers.com simply copy the digital certificate from https://www.mybank.com to pose as the bank? In other words, how is it possible to create a signature that is digital, yet can't be copied?

The answer is mathematics. This isn't a math blog, and I wont go into the math here, however, digital signatures can be thought of like this:

1. Let's pretend that every website in the world uses transparent lockable boxes to send their information to end users (sort of like the ubiquitous Amazon cardboard boxes but obviously these boxes are transparent). The information inside each lockable box is printed on a postcard (one postcard per box) that is completely visible through the transparent box.

2. Each website that sends information in a box requires the end user / web browser to open the box with a key that is unique to the website sending the information. This key is not kept by the website / box sender, but is instead kept by an authority that can be trusted to supply the right key for each website's boxes on demand. The trusted authority is necessary because it is possible that the website / box sender can't be trusted to supply the correct key in situations where the sender does not want to be held accountable for the content in the box!

3. Upon receiving a transparent lockable box from a website, an end user / web browser will ask the authority for the key belonging to the website that is claiming to have sent the box. If the key provided opens the box, the information on the postcard contained within the box is guaranteed to be from the website claiming to have sent it (for example mybank.com). If the key provided does not open the box, then the information on the postcard could be from somewhere else (for example evilhackers.com) and should not be trusted. Without successfully opening the transparent box with the key, the end user / web browser has no certainty of the origin of the postcard within.

In reality, this process is achieved using analogous mathematical objects that can be represented digitally. The padlocks and keys in real life are represented by numbers, and the act of locking and unlocking the boxes is represented by multiplication using clock arithmetic. The postcard in each box represents one of the HTTP verbs and some associated information.

The situation where a box cannot be unlocked, because the key is not present or is incorrect, is a situation many users may have seen before. It results in a browser message that looks something like the following:

I've simulated this message by pointing the web browser at my local server, however the same message would show for a site that had an invalid or expired certificate. 

HTTPS Benefit 2: Encryption of information (public key cryptography)

The second security benefit that HTTPS provides is the encryption of all information transmitted over the HTTPS connection. This encryption is necessary, because as in our previous analogy, our boxes while guaranteed to be from the sender (provided the key works), are still transparent and contain only the equivalent of a postcard that anyone can read.

Also, once the boxes are on the internet, there are no guarantees about where those boxes may end up. In short, anyone could read their contents.

You may be thinking that this encryption is an easy problem to solve. If both the website and the end user could agree on a secret code in advance, the website and the end user could both encrypt and decrypt the information sent to each other using the secret code, much like the Germans did with the Enigma machine. However, this wont work on the internet, because the initial agreement about which secret code to use is still subject to being listened into. The real question is: How can we encrypt a secret message, write it on a postcard, and then send it to someone we don't know so that they can still read it? If we do encrypt the message, the postal workers that deliver it wont be able to read the postcard, but the intended recipient wont be able to read it either!

HTTPS uses something called public key cryptography to encrypt the messages being sent back and forth between the website and the end user / web browser. The major strength of this technique is that it does not require the communicating parties to get together and agree on a secret code in advance.

Instead a 'shared secret' is developed between the communicating parties, often using an algorithm called the Diffie-Hellman key exchange. This is an ingenious way of creating a 'shared secret' in a public space using a 'public key' and a 'private key'. You can read more about the Diffie-Hellman algorithm here. There is also a simplified explanation below.

Creating a shared secret code with a stranger. It's all about public and private keys.  

Again, without getting into mathematics, the Diffie-Hellman key exchange is analogous to mixing paint.

1. To determine a shared secret code in a public place, each party selects two colours of paint, a 'public colour' and a 'private colour'. The public colours are shared publicly with anyone who wants to see them. The parties keep their private colours to themselves and they are never ever shared.

2. After selecting paints, the next thing both parties do is create a 'public private mixture', which is a publicly shared mixture of both their private and public colours. Anyone can access this mixture, but having the mixture still does not reveal the private colours, because colours cannot be decomposed after they have been mixed (it's a one way mixture).

3. To get a shared secret (or in this case a shared secret colour), the final step is for each party to take a portion of the other parties public private mixture to their own private space, and add their private colour to it. The magic here is that both parties will then end up with the same colour, however anybody 'eavesdropping' on their public conversation will still not know what the shared colour is because the shared colour includes private colours that have never been shared.

In real life, the public and private colours are known as public and private keys. Each pot of paint represents a number, and 'mixing' represents a one way cryptographic function.

What does encrypted information look like?

Always with HTTP, a client such as a web browser sends a request to a server that then returns a response. Therefore it can be seen that there are two parts to each request. A request, and then its associated response. HTTP and HTTPS always include both, so both the request and the response are below. Before we can appreciate encrypted information though, it probably helps to see what unencrypted information looks like. Therefore, an unencrypted request and response is shown below.

An unencrypted HTTP REQUEST (with only the very sensitive parts of the request highlighted in red)

GET http://localhost/ HTTP/1.1Host: localhostConnection: keep-aliveCache-Control: max-age=0Authorization: Basic <password was here but I've intentionally commented it out>Upgrade-Insecure-Requests: 1User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8Accept-Encoding: gzip, deflate, brAccept-Language: en-US,en;q=0.9Cookie: _ga=GA1.1.2047538122.1539517159; JSESSIONID=E293BB3768AD3AC6DFB57B48DAA489A6 (enough information to hack us through session hijacking)

An unencrypted HTTP RESPONSE (with very sensitive parts of the request highlighted in red)

HTTP/1.1 200X-Content-Type-Options: nosniffX-XSS-Protection: 1; mode=blockCache-Control: no-cache, no-store, max-age=0, must-revalidatePragma: no-cacheExpires: 0X-Frame-Options: DENYAccess-Control-Allow-Origin: *Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETEAccess-Control-Max-Age: 3600Access-Control-Allow-Headers: x-requested-withContent-Type: text/html;charset=UTF-8Content-Language: en-USTransfer-Encoding: chunkedDate: Sat, 20 Oct 2018 18:14:07 GMTHTTP/1.1 200X-Content-Type-Options: nosniffX-XSS-Protection: 1; mode=blockCache-Control: no-cache, no-store, max-age=0, must-revalidatePragma: no-cacheExpires: 0X-Frame-Options: DENYAccess-Control-Allow-Origin: *Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETEAccess-Control-Max-Age: 3600Access-Control-Allow-Headers: x-requested-withContent-Type: text/html;charset=UTF-8Content-Language: en-USDate: Sat, 20 Oct 2018 18:14:07 GMTContent-Length: 471<!DOCTYPE HTML><html><head> <title>HTTPS Service</title> <meta http-equiv="Content-Type" content="text/html; charset=UF-8" /></head><body><h1>Administration panel</h1><h3>Calculator service</h3><h4>Let's Encrypt SSL</h4><p>The current SSL certificate expires in <span>0</span> days.</p><p>Restarting this service less than 30 days before the certificate expiry will automatically enew the current certificate.</p></body></html>

A small discussion about the unencrypted HTTP request and response

The HTTP request and response have their very sensitive data highlighted in red. The parts of the request highlighted in red are sensitive because they are credentials that would allow a hacker to pretend to be a victim.

The highlighted red text in the response is sensitive, firstly because it shows what the user is looking at, but secondly because it can be modified by a 3rd party / evil hacker before the end user sees it. This gives an evil hacker the ability to embed malicious content, links or scripts in the users page without the user knowing about it.

An encrypted HTTPS request

Although the HTTPS request and response contain the same data as the HTTP request and response above, there is no information that would be useful to a hacker shown in the HTTPS request and response below. This is because the HTTPS request and response below are both encrypted. 

CONNECT api.quotecrunchers.com:443 HTTP/1.1
Host: api.quotecrunchers.com:443
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36

A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.

Version: 3.3 (TLS/1.2)
Random: 51 0E 90 03 B0 0A F7 C6 BC 8B 32 A3 1C AD C4 B8 DC 2A 8F 9D 9A D0 7E 3A 84 61 80 90 87 CE 39 26
"Time": 24/11/1971 8:28:17 AM
SessionID: 9B E7 15 7E 54 9D 26 96 B3 4C 91 0D 7C 41 81 B3 9D 20 44 83 E5 64 BF 8E E1 4F D8 B5 32 DA 3A FD
Extensions:
0xfafa empty
renegotiation_info 00
server_name api.quotecrunchers.com
extended_master_secret empty
SessionTicket empty
signature_algs sha256_ecdsa, Unknown[0x8]_Unknown[0x4], sha256_rsa, sha384_ecdsa, Unknown[0x8]_Unknown[0x5], sha384_rsa, Unknown[0x8]_Unknown[0x6], sha512_rsa, sha1_rsa
status_request OCSP - Implicit Responder
SignedCertTimestamp (RFC6962) empty
ALPN h2, http/1.1
channel_id(GoogleDraft) empty
ec_point_formats uncompressed [0x0]
0x0033 00 29 2A 2A 00 01 00 00 1D 00 20 F9 10 6E E5 6A 7B FF 39 7E 12 D1 C9 8D 93 64 7E C8 88 88 3F 89 A7 EC 56 AD F4 81 97 3C 0D B3 3D
0x002d 01 01
0x002b 0A 8A 8A 03 04 03 03 03 02 03 01
elliptic_curves unknown [0x2A2A), unknown [0x1D), secp256r1 [0x17], secp384r1 [0x18]
0x001b 02 00 02
0x5a5a 00
padding 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Ciphers:
[CACA] Unrecognized cipher - See http://www.iana.org/assignments/tls-parameters/
[1301] Unrecognized cipher - See http://www.iana.org/assignments/tls-parameters/
[1302] Unrecognized cipher - See http://www.iana.org/assignments/tls-parameters/
[1303] Unrecognized cipher - See http://www.iana.org/assignments/tls-parameters/
[C02B] TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
[C02F] TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[C02C] TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
[C030] TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
[CCA9] TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
[CCA8] TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
[C013] TLS1_CK_ECDHE_RSA_WITH_AES_128_CBC_SHA
[C014] TLS1_CK_ECDHE_RSA_WITH_AES_256_CBC_SHA
[009C] TLS_RSA_WITH_AES_128_GCM_SHA256
[009D] TLS_RSA_WITH_AES_256_GCM_SHA384
[002F] TLS_RSA_AES_128_SHA
[0035] TLS_RSA_AES_256_SHA
[000A] SSL_RSA_WITH_3DES_EDE_SHA

Compression:
[00] NO_COMPRESSION

An encrypted HTTPS response

HTTP/1.1 200 Connection Established
Version: 3.3 (TLS/1.2)
SessionID: 5B CB 71 91 D7 12 52 9B D7 9E 2D 30 2B 5B C1 8B E8 D7 FC F6 78 E4 F2 93 4D 1A B7 F6 E8 70 1C B8
Random: 5B CB 71 91 83 53 43 25 8A C6 80 F4 F1 17 E6 F6 62 7E 05 4A 48 2A CC CB 0E AA 02 FE FD 21 AE C3
Cipher: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 [0xC02F]
CompressionSuite: NO_COMPRESSION [0x00]
Extensions:
renegotiation_info 00
extended_master_secret empty

It can be seen that there is no information in the HTTPS request or response that would be useful to a hacker. All data has been encrypted so that it is unrecognisable, which is of course the point of encryption with HTTPS.

Summary

We've had a high level overview of HTTPS and the security benefits it provides.

We've covered the major components of HTTPS including:

• Digital signatures
• Certificate authority
• Public key encryption

In the next blog post, we'll look at some specifics around how we can use this information to build HTTPS into a software application.

Projects: When the metal hits the road

I like projects. Projects are where the metal hits the road, where all the theory that you've read and all the learning that you've done gets truly tested.

Sometimes everything works out and all is according to plan. We all know however that that is not always the case.

A colleague has a favorite quote, and I like it too. Hardware eventually fails, Software eventually works.

I have a bunch of projects I'm actively working on at the moment, software projects, not hardware projects. I hope that they will eventually work. I've listed what I think are my most interesting current projects below.

1. Auto-renewing SSL integrated into spring-boot

HTTPS is taking over the world at the moment (or rather the software industry is forcing HTTPS on everyone which is great). Given the challenge of needing to add HTTPS to basically everything, this project is to create some generic reusable code, that can be used inside spring-boot applications to create immediate certificate authority compliant SSL connections.

2. Single sign-on

This is a project to allow the users of a large SaaS application to access all services they're authorized to access with a single set of credentials. It is complicated by the fact that the original system has already been in production for several years, and many users already use different accounts to access different parts of the system. As well as the creation of new accounts, existing accounts and their permissions will need to be merged for this project to be a success. 

3. A gig and beer guide

For quite some time I've been working on a gig and beer guide, although the description of a gig and beer guide does not adequately capture all that this is about. It's more of a singing, dancing all encompassing social network for musicians to get in touch with venues and plan gigs and advertise events all around the world.  It's always been a bit of an R&D project, and this is an attempt to build a slimmed down (releasable!) version of it.

I will write about these projects and others in subsequent blog posts.

(Yet another) Information technology blog

An introduction

The longer I spend in the information technology field, the more meta I get in my thinking. 

Instead of focusing on the very black and white specifics of a language or a technique in isolation, I more often find myself thinking more about the grey areas. What are current trends, how do I sell my ideas to a team, how do I determine how much 'gold plating' is required for a feature, what are the pros and cons of a certain process or approach to solving a larger problem, is it better to build vs buy, and is it better to host on premise or in the cloud?

Often, there are no hard and fast answers to these sorts of questions. With an abundance of technical blogs offering specific technical solutions to just about anything one can think of, I figure with my blog, I'll go the other way, and explore the grey.

Is this a good solution? Could there be a better solution? What makes a good solution anyway? I certainly don't have the answers, however I may have a thought or two, so this blog will hopefully be those. 

New Zealand really is a beautiful country

   Flying in over the Manakau harbour this morning on the flight in from Singapore, I was really struck by how beautiful New Zealand really is.

Last post from foreign soil

Well, this is my last post from outside New Zealand. I've spent the last 5 hours or so exploring Changi airport in Singapore. I've seen the butterly house, visted the cactus garden, and had a swim at the rooftop pool. In a few minutes I'll be filling out my declaration form for New Zealand, with more questions than all the other countries I have visted combined.

Are you bringing in any...

food (yes)
plants or plant products including bamboo (yes)
Animals or animal products (yes)
Carrying items that have been used outdoors (yes)
Carrying items on behalf of anybody else (yes)
Potentially prohibited medications (yes - it's all in Russian, I have no idea what it is)
Had any contact with farm animals within the last 30 days (yes)

There is a chance that customs could take a while...

I'm really looking forward to seeing everyone back in New Zealand.

Heading home and post trip updates

I am now in Singapore airport and heading home. I have had an amazing time travelling through Asia, but right now, I am looking forward to getting back to Wellington, getting back to work, and sleeping in my own bed (which will now have new lovely silk bedding from Beijing)

There are some more places I wanted to see that I missed out on due to time restrictions. Notably Chaing Mai, Pai, Vietnam, Laos and South China. They will have to be another trip.
Over the next week or so, in order to provide a more complete record of where I did get to, I plan to include the stories and photos that I missed out on because I was too busy travelling.

Mum, if you're reading this, I am arriving on flight NZ445 at 16:30 into Wellington on Monday the 27th. I have very heavy bags (I've maxed out my weight) as they are full of NZ correspondence school work from Liz's family in Mongolia which was the least I could do for them putting me up. The travel agent told me I would be returning on Sunday the 27th, but after having checked my callender I now realise there was some confusion and the 27th is a Monday.

It will take me a while to get through customs so I am hoping you can meet me after you finish work.

See you in New Zealand!

What I would do differently next time

What I would do differently next time

1) Buy my electronics in New Zealand. This is because brand name electronics are the same price or cheaper in New Zealand, Electronics purchased in New Zealand are guarunteed to have warranties that are servicable in New Zealand and buying electronics in foreign countries in languages other than English is hard work and time consuming.

2) Bring a netbook This is because free wifi hotspots are ubiquitous. Internet cafes are not and finding them eats up a lot of time, especially in countries where there is very little English spoken. Also, at times Internet cafe's can be very pricy (i.e. $10NZ for 20 minutes at Beijing Airport).

3) Always check out the local scams before arriving somewhere (aka read lonely planet - online travel forums etc). On several occasions the lonely planet had details of scams and how to avoid them, but I fell into them because I had not read the perfectly good information I had armed myself with. At other times, I had read the information which saved me from some potentially expensive encounters.

4) Always know how much I am spending before agreeing to services and get it in writing. This sounds extreme, but using a paper and pen as a communication tool to negotiate price is good because it keeps a record of what was agreed and keeps the seller honest. If a price is only agreed verbally, sellers (often street vendors and taxi drivers) will hike the price once goods are received. Paper and pen is good because it keeps a record, so no vendor can inflate the price post sale.

5) Use a GPS from the start, and buy the appropriate local maps before leaving New Zealand or whatever country (where possible). The GPS has been amazing. I never get lost, always know where I'm going, I can meander anywhere with confidence because I'll always be able to get back. The GPS (with appropriate maps) also points out local tourist attractions for me so I don't end up walking past stuff, and it provides an odometer on taxi rides that has allowed me to calculate the rate exactly owed to taxi drivers (and not a cent more) since I bought it. On more than a few occasions since I got the GPS, taxi drivers have attempted to charge me inflated prices, but in every instance they have had to accept my odometer reading and take what I gave them (as I am not flexible when it comes to taxi drivers). One taxi driver in Mongolia got stroppy and one grabbed my shirt to stop me getting out of his cab without paying the $5 US per kilometer he wanted, but fortunately he let go, I leapt out, and threw his opportioned money on the ground (about $2 NZ) so he had to scramble for it if he wanted it and by that time I was gone. I no longer have any patience for fraudsters.

6) Always check out the local public transport options before booking a tour or at least understand there is always a local cheap option. On more than a couple of occasions I'd pay something like $30 USD for a tour somewhere to find that entrace was free and somebody else was smarter and had bought a local bus ticket for fifty cents. Once in Cambodia I paid for a  'tour', and they put me on the local bus everyone else was paying fifty cents for anyway. (they didn't provide any other services).