Founder and CEO of @TentamenHr. Organizer of Zagreb Software Testing Club meetups.
1441 stories
·
0 followers

What’s new in Elixir – Nov/18

1 Share

Hi everyone!

We are back with another edition of “What’s new in Elixir”. This time we are working towards Elixir v1.8 which should be out next January. Here are some highlights of what have already made to master:

  • mix test now supports the --max-failures flag (commit)

  • The Inspect protocol can now be easily derivable and this is a great mechanism to exclude some fields during inspection and logs, especially security-related ones (commit)

  • Mix now automatically starts optional dependencies when using the :extra_applications configuration (commit)

  • We finally have official support for time zone databases in Elixir. Elixir still handles only UTC by default but you can bring another database to use the DateTime API with other time zones (commit)

  • There are many other small additions, such as List.myers_difference/3 and String.bag_distance/2 which were extracted from the Elixir codebase itself

For the complete list of changes, see the CHANGELOG. To keep track of upcoming changes and even contribute some features yourself, head to Elixir’s issues tracker.

banner-elixir development subscription

Read the whole story
karlosmid
18 hours ago
reply
Zagreb
Share this story
Delete

Building a new MySQL adapter for Ecto, Part I: Hello World

1 Share

As you may have seen in the announcement, Plataformatec is working on a new MySQL driver called MyXQL.

Writing a complete driver involves quite a bit of work. To name just a few things, we need to support: all protocol messages and data types, authentication schemes, connection options (TCP/SSL/UNIX domain socket), transactions and more. Rather than going through all of these in detail, I plan to distill this knowledge into 4 parts, each with a quick overview of a given area:

  • Part I: Hello World (this article!)
  • Part II: Encoding/Decoding
  • Part III: DBConnection Integration
  • Part IV: Ecto Integration

This also mimics how I approached the development of the library, my end goal was to integrate with Ecto and I wanted to be integrating end-to-end as soon and as often as possible. Rather than implementing each part fully, I implemented just enough to move forward knowing I can later go back and fill in remaining details. Without further ado, let’s get started!

Hello World

Our “Hello World” will involve performing a “handshake”: connecting to a running MySQL server and authenticating a user. To avoid getting bogged down in authentication details, the simplest possible thing to do is to log in as user without password. Let’s create one:

$ mysql --user=root -e "CREATE USER myxql_test"

We can check if everything went well by trying to log in as that user:

$ mysql --user=myxql_test -e "SELECT NOW()"
+---------------------+
| NOW()               |
+---------------------+
| 2018-10-04 18:35:11 |
+---------------------+

If you don’t have MySQL installed, I recommend setting it up via Homebrew, if you’re on macOS, or Docker. I ended up using Docker because I knew I needed to test on multiple server versions. Here’s how I set it up:

$ docker run --publish=3306:3306 --name myxql_test -e MYSQL_ROOT_PASSWORD=secret -d mysql:8.0.12
# note we connect via TCP, instead of the default UNIX domain socket:
$ mysql --protocol=tcp --user=root --password=secret -e "CREATE USER myxql_test;"

$ mysql --protocol=tcp --user=myxql_test -e "SELECT NOW()"
+---------------------+
| NOW()               |
+---------------------+
| 2018-10-04 18:40:04 |
+---------------------+

We can now connect to the server from IEx session:

iex> {:ok, sock} = :gen_tcp.connect('127.0.0.1', 3306, [:binary, active: false], 5000)
{:ok, #Port<0.6>}

Let’s break this down. :gen_tcp.connect/4 accepts:

  1. Hostname (as charlist)
  2. Port
  3. Options (as proplist); by default, data from the socket is returned as iolist, however for us binary will be more convenient to work with, so we pass :binary option.
    active: false means we’ll work with the socket in “passive mode”, meaning we’ll read data using blocking :gen_tcp.recv/3 call.
  4. Timeout (in milliseconds)

Let’s now read data from the socket: (0 means we read all available bytes, 5000 is the timeout in milliseconds)

iex> {:ok, data} = :gen_tcp.recv(sock, 0, 5000)
iex> data
<<0, 0, 0, 10, 56, 46, 48, 46, 49, 50, 0, 12, 0, 0, 0, 11, 9, 19, 27, 96, 108, 77, 116, 0, 255, 255, 255, 2, 0, 255, 195, 21, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 37, 62, 29, 59, 1, ...>>

To make sense of this, we’re gonna need to look into MySQL manual.
Each MySQL packet has 3 elements: length of the payload (3-byte integer), sequence id (1-byte integer), and payload.
In this case, the actual payload is the “Initial Handshake Packet”. Let’s extract the payload part using binary matching (see <<>>/1 for more information on binary matching):

iex> <<payload_length::24, sequence_id::8, payload::binary>> = data
iex> payload_length
4849664
iex> byte_size(payload)
74

Wait, the size of the payload is 74 so why payload_length is 4849664?! Numerical values when stored in a binary have “endianness” which basically means whether we should read bits/bytes from “little-end” (least significant bit) or “big-end” (most significant bit).
Thus, a 3-byte integer <<74, 0, 0>> in “big-endian” is indeed 4849664 but in “little-endian” it’s 74. Fortunately, bitstring syntax has great support for endianess and it’s as easy as adding little modifier (“big-endian” is the default):

iex> <<payload_length::24-little, sequence_id::8, payload::binary>> = data
iex> payload_length
74

To make sense of the remaining payload we’re gonna use the binpp package:

iex> :binpp.pprint(payload)
0000 0A 38 2E 30 2E 31 32 00 0F 00 00 00 27 73 79 59  .8.0.12.....'syY
0001 7A 34 26 3B 00 FF FF FF 02 00 FF C3 15 00 00 00  z4&;.ÿÿÿ..ÿÃ....
0002 00 00 00 00 00 00 00 43 55 6B 60 74 5A 71 08 75  .......CUk`tZq.u
0003 6F 08 2F 00 63 61 63 68 69 6E 67 5F 73 68 61 32  o./.caching_sha2
0004 5F 70 61 73 73 77 6F 72 64 00                    _password.

We can see up to 16 bytes in each row and at the far right we have ASCII interpretation of each byte. Per “Initial Handshake Packet” the first byte is the protocol version, always 10 (0x0A), and what follows is a null-terminated server version string. Let’s extract that:

iex> <<10, rest::binary>> = payload
iex> [server_version, rest] = :binary.split(rest, <<0x00>>)
iex> server_version
"8.0.12"

We can parse the server version, that’s a good start! There are other fields in this packet that in a complete adapter we’d have to handle, but for now we’ll simply ignore them. We’ll just take a note of the authentication method at the end to the packet, a null-terminated string "caching_sha2_password".

After receiving “Initial Handshake Packet” the client is supposed to send “Handshake Response”. We’ll again just gloss over the details:

iex> use Bitwise
iex> capability_flags = 0x00000200 ||| 0x00008000 ||| 0x00080000
iex> max_packet_size = 65535
iex> charset = 0x21
iex> username = "myxql_test"
iex> auth_response = <<0x00>>
iex> client_auth_plugin = "caching_sha2_password"
iex> payload = <<
       capability_flags::32-little,
       max_packet_size::32-little,
       charset, 0::8*23,
       username::binary, 0x00,
       auth_response::binary,
       client_auth_plugin::binary, 0x00
>>
iex> sequence_id = 1
iex> data = <<byte_size(payload)::24-little, sequence_id, payload::binary>>

Let’s break this down:

First, we use CLIENT_PROTOCOL_41,CLIENT_SECURE_CONNECTION, and CLIENT_PLUGIN_AUTH capability flags using “bitwise OR”. Secondly, we set the max packet size, charset (0x21 is utf8_general_ci), filler (0s repeated 23 times), username, auth response (empty password is a null byte), and auth plugin name. Note, we encode username and client_auth_plugin as null-terminated strings. Finally, we generate payload and encode it in a packet with payload length and sequence id (it’s 2nd packet so sequence id is 1). Let’s now send this and receive response from the server:

iex> :ok = :gen_tcp.send(sock, data)
iex> {:ok, data} = :gen_tcp.recv(sock, 0)
iex> <<payload_length::24-little, sequence_id::8, payload::binary>> = data
iex> :binpp.pprint(payload)
0000 00 00 00 02 00 00 00

The first byte of the response is 0x00 which corresponds to the OK_Packet, authentication succedded! Even though we’ve glossed over many details, we’ve shown that we can integrate with the server end-to-end and that’s going to be a foundation we’ll built upon. There are many more packets that we’ll need to encode or decode and we’re gonna need a more structured approach which we will discuss in part II.

banner-elixir development subscription

Read the whole story
karlosmid
19 hours ago
reply
Zagreb
Share this story
Delete

Verifying request signatures in Elixir/Phoenix

1 Share

Comments

Read the whole story
karlosmid
19 hours ago
reply
Zagreb
Share this story
Delete

Nested Layouts With Phoenix

1 Share

Over the last few weeks, we have been building a web application in one of our clients and ended up duplicating some template code. These new pages had something in common between them, but not with the rest of the application. We needed an inner layout to reuse the template code between these pages, however, Phoenix doesn’t come with this feature. In this post, you’ll learn how you can build nested layouts in Phoenix and when you should use them.

Why are they useful?

Inner (or nested) layouts are useful to reuse template code when you have subsections on your website. Usually, in Phoenix applications, we have /templates/layout/app.html.eex that shares template code between all pages. For example:

Example of Layout

 

In the example above, we want to reuse the application logo, title, and navigation menu across all pages. We can solve that by using the Phoenix default layout and it works great. However, sometimes you need to create a new website section inside the parent layout. For example, imagine we want to add a help section to our Phoenix website. Look at the image below:

Inner Layout

We still want to reuse the layout header across all pages, but we also want to reuse the left navigation and main content layout in all pages inside the help section. Let’s see how we can do that using Phoenix.

The Inner Layout Solution

I tried some solutions by looking at some examples online. After discussing with the Plataformatec team, we came up with an approach that’s very simple and extensible, thanks to Phoenix explicit layout rendering. The solution works like this:

layout template

  • You create the nested layout template.
  • In the nested layout, you set the parent template by calling a function.
  • The parent layout is aware that is possible to have an inner layout and will render it.
  • You can invoke your nested layout by using the Phoenix `put_layout/2` function.

The main goal here is to make the nested layout work like any Phoenix layout. This will make it familiar to any Phoenix developer.

 Preparing The Parent Template

First, let’s add a function that allows a template to render the parent layout:

# views/layout_view.ex
defmodule YourAppWeb.LayoutView do
  use YourAppWeb, :view

  def render_layout(layout, assigns, do: content) do
    render(layout, Map.put(assigns, :inner_layout, content))
  end
end

The render_layout/3 will render the given layout by assigning the contents of the inner layout given in the do argument. Now, in the parent layout, we need to render the inner layout contents or the controller view contents. Look at how you can do it:

<%# templates/layout/app.html.eex %>

<%# ... %>

  <%= Map.get(assigns, :inner_layout) || render @view_module, @view_template, assigns %>

<%# ... %>

With the code above, our parent layout is aware that there may be an inner layout and it should render its contents when available. That’s it! Now let’s see how we can use it.

Using the Nested Layout

You can create a nested layout template in the layouts folder and use it like this:

<%# templates/layout/nested_layout.html.eex %>

<%= render_layout "app.html", assigns do %>
  <%# Your HTML markup here %>
  <%= render @view_module, @view_template, assigns %>
  <%# More HTML markup here %>
<% end %>

We render the parent and put the inner content in the do/end blocks in our nested layout template. Any content outside of the do/end blocks will not be rendered. Don’t forget to call render @view_module, @view_template, assigns, or the contents of your controller’s action template will not be rendered.

Now you can use our nested layout by invoking the plug put_layout/2 in your controller or router. For example:

defmodule YourAppWeb.NestedContentController do
  use YourAppWeb, :controller

  plug :put_layout, :nested_layout

  def index(conn, params) do
    # stuff
  end
end

You can organize your nested layout files in a different way. For example, imagine you have a help section in your website and you want to keep the nested layout file in the help folder. In your HelpView you’ll need to import the render_layout/3 function, like this:

# views/help_view.ex

defmodule YourAppWeb.HelpView do
  use YourAppWeb, :view
  import YourAppWeb.LayoutView, only: [render_layout: 3]
end

After that, you can put your nested layout template in templates/help/layout.html.eex. In the controller or router, you can invoke the nested layout like this:

plug :put_layout, {YourAppWeb.HelpView, "layout.html"}

The code above will render your layout.html.eex template in views/help directory using YourAppWeb.HelpView module.

Wrapping Up

Inner layouts come in handy to reuse template code of subsections of your web application. You learned how simple it is to build that in Phoenix. Just be aware of not creating inner layouts of inner layouts. If you do that, your codebase will start to be very hard to maintain. You can see and try by yourself a sample Phoenix app running the inner layout example.

Have you implemented inner layout in a different way? Do you have any feature that you would like to see how we can build it in Phoenix? Let us know in your comments.

Read the whole story
karlosmid
19 hours ago
reply
Zagreb
Share this story
Delete

Phoenix 1.4.0 released

1 Share
Comments
Read the whole story
karlosmid
3 days ago
reply
Zagreb
Share this story
Delete

Most advanced XSS detection suite

1 Share
Comments
Read the whole story
karlosmid
3 days ago
reply
Zagreb
Share this story
Delete
Next Page of Stories