Languages

This article is not set out to try and persuade you, the reader, that using an iterator or materialized collection will universally solve your problems. Iterator and materialized collection usage can be used to solve the scenarios that we’ll be looking at, but both will come with a different set of pros and cons that we can further explore. The purpose of this article is to highlight scenarios based on real world experiences where an iterator or materialized collection was being misunderstood, misused, and, ultimately, leading to a pile of headaches. As you read this article, if you find yourself saying “Well, sure, but they should have…” you’re probably right. The problem is fundamentally not the usage of an iterator or the materialized collection, but not understanding how to consume them effectively. So, I hope that when you’re working with newer software engineers or people less familiar with some of these concepts, you can be reminded to impart your wisdom. If you’re interested in the companion video to this article, check out this YouTube video: “Hidden Dangers of Iterators and Collections in C#.” Common Iterator and Collection Scenario Setup To give us some common ground as we explore an approach with a materialized collection in contrast with an iterator, let’s expand on the real world examples where I see these challenges regularly coming up. Let’s assume that you have a data access layer in your application that is responsible for getting records from a database or some data store. You build some API that the rest of your application can use, and you’ll be using the results of that API in situations such as: Running LINQ methods (Any(), Count(), or even filtering using Where()). Showing resulting data sets in a user interface. Using resulting data for sorting, filtering, or otherwise running algorithms with this data as the source. Another key ingredient to mention here is that because this is anchored in the real world…code bases change and evolve over time. People come up with new use cases for the data access layer. There’s more data added into the data store pushing limits that people never would have accepted. You have new or more junior developers coming into the code base. It’s real life and until we have more automated tech to police these things, we’re going to run into fun issues. Materializing Large Datasets Before we focus on iterators, let’s explore the more common approach, which involves materialized collections. Given the common scenario we discussed above, you have a method that might look like the following: public List<string> GetEntriesFromDatabase() { // incur some latency for connecting to the database var connection = _connectionFactory.OpenNew(); var command = connection.CreateCommand(); // TODO: actually create the query on the command, but this is just to illustrate var resultsReader = command.Execute(); List<string> results = new List<string>(); while (resultsReader.Read()) { // TODO: pull the data off the reader... this example just uses a single field var value = resultsReader.GetValue(0); results.Add(value); } return results; } There is nothing glaringly wrong with this example, and, in fact, by leaving the actual query up to your imagination, I’ve omitted where a lot of the trouble can come from. Let’s use an example from my GitHub repo to simulate what this might look like so we have a reference point: List<string> PretendGetEntriesFromDatabase() { // let's simulate some exaggerated latency to the DB Thread.Sleep(5000); Console.WriteLine($"{DateTime.Now} - <DB now sending back results>"); // now let's assume we run some query that pulls back 100,000 strings from // the database List<string> results = new List<string>(); while (results.Count < 100_000) { // simulate a tiny bit of latency on the "reader" that would be // reading data back from the database... every so often we'll // sleep a little bit just to slow it down if ((results.Count % 100) == 0) { Thread.Sleep(1); } results.Add(Guid.NewGuid().ToString()); } return results; } Note: the delays in the example code above are artificially inflated so that if you run this in a console, you can observe the different effects of changing the variables. Now that we have the code snippet that simulates pulling from the database by building up a full collection first, let’s look at some calling code that can exercise it (also on GitHub): long memoryBefore = GC.GetTotalMemory(true); Console.WriteLine($"{DateTime.Now} - Getting data from the database using List..."); List<string> databaseResultsList = PretendThisGoesToADatabaseAsList(); Console.WriteLine($"{DateTime.Now} - Got data from the database using List."); Console.WriteLine($"{DateTime.Now} - Has Data: {databaseResultsList.Any()}"); Console.WriteLine($"{DateTime.Now} - Count of Data: {databaseResultsList.Count}"); long memoryAfter = GC.GetTotalMemory(true); Console.WriteLine($"{DateTime.Now} - Memory Increase (bytes): {memoryAfter - memoryBefore}"); The calling code will take a snapshot of memory before we call our method and perform operations on the result. The two things we’ll be doing with the result are: Calling the LINQ method Any(). Calling Count directly on the list. As a side note, the Count() LINQ method will not require full enumeration as it has an optimization to check if there’s a known length. Examining the Materialized Collection Results With the materialized collection example, we can call the method and store the result set in memory. Given the two operations we are trying to use on the collection, Any() and Count, this information is quickly accessible to us because we’ve paid the performance hit one time to materialize the results into a list. Compared to an iterator, this approach does not run the risk of allowing callers to accidentally fully re-enumerate the results. This is because the result set is materialized once. However, the implication here is that, depending on the size of the results and how expensive it might be to fully materialize that full result set, you could be paying a disproportionate price for things like Any() that only need to know the existence of one element before they return true. If you recall what I said at the start of this article: if your mind automatically jumps to “Well someone should build a dedicated query for that,” then…yes, that’s absolutely a solution. But, what I’m hear to tell you is that it’s a very common thing for something like this to slip through the cracks of a code review because of the LINQ syntax we have available to us. Especially if someone rights something like: CallTheMethodThatActuallyMaterializesToAList().Any() In this example, if the method name wasn’t quite so obvious, you’d have no issue with an iterator but a huge concern with a heavy-handed list materialization. Why is it so heavy-handed? Well, one could argue it’s doing exactly what it was coded to do, but we need to consider how callers are going to be taking advantage of this. If callers rarely ever need to be dealing with the full data set and they need to do things like Any(), First() or otherwise lighter weight operations that don’t necessarily need the entire result set…They don’t have a choice with this API. They will be paying the full price to materialize the entire result set when, in reality, maybe they just needed to walk through several elements. In the example code above, this results in multiple megabytes of string data being allocated when we need a count of data and to check if there was any data. Yes, it looks contrived, but this is simply to illustrate that this API design does not lend itself well to particular use cases for callers. Let’s Look at Iterators Let’s go ahead and contrast the previous example with an iterator approach. We’ll start with the code, which you can find on GitHub: IEnumerable<string> PretendThisGoesToADatabaseAsIterator() { // let's simulate some exaggerated latency to the DB Thread.Sleep(5000); Console.WriteLine($"{DateTime.Now} - <DB now sending back results>"); // now let's assume we run some query that pulls back 100,000 strings from // the database for (int i = 0; i < 100_000; i++) { // simulate a tiny bit of latency on the "reader" that would be // reading data back from the database... every so often we'll // sleep a little bit just to slow it down if ((i % 100) == 0) { Thread.Sleep(1); } yield return Guid.NewGuid().ToString(); } } As you can see in the code above, we have an iterator structured to be almost identical except for: It’s an iterator. The yield return keyword is required here. The return type is IEnumerable<T> instead of List<T>. As a quick recap, an iterator will not be able to provide a caller with a count like we could do with other collection types, and all it can do is allow a caller to step through item by item. We can use a similar calling code snippet over our iterator, but let’s go ahead and add in a couple of additional console writing lines (here on GitHub): long memoryBefore = GC.GetTotalMemory(true); Console.WriteLine($"{DateTime.Now} - Getting data from the database using iterator..."); IEnumerable<string> databaseResultsIterator = PretendThisGoesToADatabaseAsIterator(); Console.WriteLine($"{DateTime.Now} - \"Got data\" (not actually... it's lazy evaluated) from the database using iterator."); Console.WriteLine($"{DateTime.Now} - Has Data: {databaseResultsIterator.Any()}"); Console.WriteLine($"{DateTime.Now} - Finished checking if database has data using iterator."); Console.WriteLine($"{DateTime.Now} - Count of Data: {databaseResultsIterator.Count()}"); Console.WriteLine($"{DateTime.Now} - Finished counting data from database using iterator."); long memoryAfter = GC.GetTotalMemory(true); Console.WriteLine($"{DateTime.Now} - Memory Increase (bytes): {memoryAfter - memoryBefore}"); The additional lines of console writing just provide some additional context for where our code will be spending time. Do Iterators Make All the Problems Go Away? The short answer: no. The long answer: Iterators can make some of the earlier issues we saw with materialized collections go away, but they come with their own challenges for folks that are not familiar with working with them. When we consider the memory footprint in this example, it’s nearly nothing in comparison to the prior example. This is the case because at no point in time in this calling code example did we need to have the entire result set materialized for us to answer the questions that we were interested in. Will that always be the case? Absolutely not. However, one of the benefits of iterators here is that a caller now has the choice. These choices include whether they just want to do partial enumeration, full enumeration, or full enumeration to materialize the result set. The key here is flexibility in how the API is consumed. But flexibility comes with a trade, and this is something I see far more frequently with newer C# programmers because they are not actually familiar with iterators. The example above? Sure, it doesn’t use much memory at all but it will run PretendThisGoesToADatabaseAsIterator twice. Yes, to you reader with the keen eyes, you likely already noticed this but with a small adjustment to the naming and calling convention: var results = GetEntriesFromDatabase(); var any = results.Any(); var count = results.Count(); Suddenly, you can’t tell if you’re dealing with an iterator or a materialized collection. Before you shout “Well this is why we never use var!” let me tweak it once more: IEnumerable<string> results = GetEntriesFromDatabase(); var any = results.Any(); var count = results.Count(); The truth is, var doesn’t matter here because you just don’t know if GetEntriesFromDatabase() is an iterator or materialized collection. So, without getting into the weeds of a million different ways we could try and improve this, the point I would like to highlight to you is that people CAN and DO get this messed up in production code bases. All of the time. A bonus round for iterators is that given the lazy nature of how they’re evaluated, I have seen layered architectures pass the enumerable all the way to a frontend to finally have it evaluated. The result was that all of the impressive asynchronous data loading support was completely foiled because the main thread ended up being the unfortunate soul that would call the iterator. Iterators or Materialized Collections? It depends. To be crystal clear, because I mentioned it in the beginning of this article, the intention of writing all of this was not to tell you that an iterator is better or worse than a materialized collection. If you’re a junior software engineer working in C#, then I hope you saw some of the trade-offs and situations where people can get caught up. Some of this stuff isn’t totally obvious and might require some practice using either approach to understand it better. If you’re a more senior software engineer and you read this article being frustrated that you had ways to solve my examples…good. I would like you to take that energy to the team you’re on and ensure you can work with more junior engineers. Help them understand where some of these issues come up and how they can avoid them. My personal preference? I like using iterator-based APIs because I like having the flexibility to stream results. However, after many years of doing this, I am digging into some of the performance characteristics. Especially when we have access to things like spans, I might be heading back out to do a bit more research.

In any web service that receives and transmits data to and from a server, the first and last events will usually be transforming the data from the format used by the web request into the format that the web server will handle, and vice versa; these operations are called deserialization and serialization, respectively. For some web services, the thought put towards this part of the flow of data is focused solely on how to configure the serialization mechanism so it works properly. However, there are some scenarios for which every CPU cycle counts, and the faster the serialization mechanism can work, the better. This article will explore the development and performance characteristics of four different options for working with the serialization of JSON messages—GSON, Jackson, JSON-B, and Kotlinx Serialization, using both the Kotlin programming language and some of the unique features that Kotlin offers compared to its counterpart language, Java. Setup Since its first release in 2017, Kotlin has grown by leaps and bounds within the JVM community, becoming the go-to programming language for Android development as well as a first-class citizen in major JVM tools like Spring, JUnit, Gradle, and more. Among the innovations it brought to the JVM community compared to Java was the data class, a special type of class that is to be used primarily as a holder of data (in other words, a Data Transfer Object, or DTO) and automatically generates base utility functions for the class like equals(), hashcode(), copy(), and more. This will form the base of the classes that will be used for the performance tests, the first of which being PojoFoo. “Pojo” stands for “Plain Old Java Object,” signifying using only basic class types of the Java programming language: Kotlin data class PojoFoo(var fizz: String, var bizz: Int, var bazz: List<String>) { constructor() : this("", 0, emptyList()) } For those who are not familiar with the Kotlin programming language: the class has three attributes: fizz bizz bazz That contain both getter and setter functions. There are two constructors for the class: One that requires arguments for each of the attributes. One that requires no arguments and populates the attributes with default values. This second constructor is the “no-arg constructor” that is typically required by JSON serialization mechanisms. In the example above, the three class attributes are marked with the keyword var; this signifies that the attributes are mutable and can be modified at any time during the lifetime of an instance of the class. To make the attributes immutable, all that is needed is to change the designator to val, upon which the attributes will become the equivalent of final attributes in Java, and Kotlin will no longer generate a getter function for the attributes. In addition, this removes the requirement of a no-arg constructor, so that can be eliminated from the code: Kotlin data class ImmutableFoo(val fizz: String, val bizz: Int, val bazz: List<String>) The next example class—DefaultValueFoo—uses a default value for the attribute fizz. This means that, if the constructor of DefaultValueFoo is invoked and no argument is provided for fizz, then the argument will be assigned the default value: Kotlin data class DefaultValueFoo(var fizz: String = "FUZZ", var bizz: Int, var bazz: List<String>) { constructor() : this(bizz = 0, bazz = emptyList()) } Finally, the example class ValueClassFoo changes the type of attribute bizz from a plain integer to an inline class. Inline classes function as wrappers around a single “payload” value; while the Kotlin code will treat the inline class as a “genuine” class, the compiler will translate the code so that only the payload value is present. This provides for several advantages compared to simply using the payload value directly, such as enforcing a type safety for different variables, for example specifying a username and a password type—two types that would normally both be strings—for a login function. In this case, it allows for the usage of UInt: a Kotlin-exclusive class that simulates the behavior of an unsigned function, something that is not supported by default by the JVM: Kotlin data class ValueClassFoo(var fizz: String, var bizz: UInt, var bazz: List<String>) { constructor() : this("", 0u, emptyList()) } Note: the class is named as such because while inline classes are still called as such in the Kotlin documentation, they have been renamed as value classes in the actual code; the keyword inline is deprecated. The Contestants GSON Introduced in 2008 and developed by Google, GSON is one of the main options that Java users employ for conducting serialization between JSON strings and Java objects and is the preferred library to leverage in Android development thanks to the support by Google. Usage The basic usage is to construct an instance of Gson and invoke the functions Gson.toJson() and Gson.fromJson() to serialize an object and deserialize a JSON string, respectively. Working With Kotlin Surprisingly, there are no additional steps necessary to work with the four example classes; all of the code snippets provided above were from the GSON testing code. Jackson Introduced in 2009, Jackson is the other widely-used JSON serialization library—alongside GSON—and is used by default in major JVM ecosystems like the Spring framework. Usage The basic usage is to construct an instance of ObjectMapper and invoke the functions ObjectMapper.writeValueAsString() and ObjectMapper.readValue() to serialize an object and deserialize a JSON string, respectively. Working With Kotlin Unlike GSON, there is quite a bit of work that is necessary to support the Kotlin features in the example classes: Jackson does not have a native concept of deserializing classes that do not possess a no-arg constructor; if it cannot find a no-arg constructor, it will normally raise an exception. A workaround for this is to mark the parameters in the constructor with @JsonProperty so that Jackson knows which argument corresponds to which class attribute: Kotlin data class ImmutableFoo( @param:JsonProperty("fizz") val fizz: String, @param:JsonProperty("bizz") val bizz: Int, @param:JsonProperty("bazz") val bazz: List<String> ) Inline classes are not processed properly due to a difference in how Jackson computes how to conduct serialization and deserialization on a class. An advantage of these serialization libraries is that they do not normally require the creation of specialized classes to conduct the serialization and deserialization actions on a class. Instead, they compute which fields to pull values from and set via reflection; whereas GSON executes the reflection actions on the actual attribute fields within the target class, Jackson’s reflection actions are targeted on the attributes’ getter and setter functions. This is an issue with inline classes, as any function that accepts or returns an inline class is name-mangled to prevent collisions with functions that might accept the equivalent “normal” type in the JVM. Thus, serializing and deserializing classes with inline class attributes will prove problematic: // Attempting to serialize to JSON expected: <{"fizz":"FUZZ","bizz":5,"bazz":["BUZZ","BOZZ"]}> but was: <{"fizz":"FUZZ","bazz":["BUZZ","BOZZ"],"bizz-pVg5ArA":5}> // Attempting to deserialize from JSON Unrecognized field "bizz" (class com.severett.serializationcomparison.jackson.model.ValueClassFoo), not marked as ignorable (3 known properties: "fizz", "bizz-WZ4Q5Ns", "bazz"]) While there is a specialized module for Jackson— jackson-module-kotlin—which provides support for many parts of Kotlin that are not included in the testing here (e.g., Pair, Triple, IntRange, etc.), it does not provide support for inline classes and does not plan on offering support for the foreseeable future. Instead, it is necessary to create custom serializer and deserializer classes to handle ValueClassFoo and mark ValueClassFoo with @JsonSerialize and @JsonDeserialize, respectively: Kotlin class ValueClassFooSerializer : JsonSerializer<ValueClassFoo>() { override fun serialize(value: ValueClassFoo, gen: JsonGenerator, serializers: SerializerProvider?) { gen.writeStartObject() gen.writeStringField(ValueClassFoo.FIZZ_FIELD, value.fizz) gen.writeNumberField(ValueClassFoo.BIZZ_FIELD, value.bizz.toInt()) gen.writeArrayFieldStart(ValueClassFoo.BAZZ_FIELD) value.bazz.forEach(gen::writeString) gen.writeEndArray() gen.writeEndObject() } } class ValueClassFooDeserializer : JsonDeserializer<ValueClassFoo>() { override fun deserialize(jsonParser: JsonParser, ctxt: DeserializationContext?): ValueClassFoo { val node = jsonParser.codec.readTree<JsonNode>(jsonParser) return ValueClassFoo( fizz = node[ValueClassFoo.FIZZ_FIELD].asText(), bizz = node[ValueClassFoo.BIZZ_FIELD].asInt().toUInt(), bazz = (node[ValueClassFoo.BAZZ_FIELD] as ArrayNode).map { it.textValue() } ) } } @JsonSerialize(using = ValueClassFooSerializer::class) @JsonDeserialize(using = ValueClassFooDeserializer::class) data class ValueClassFoo(var fizz: String, var bizz: UInt, var bazz: List<String>) { constructor() : this("", 0u, emptyList()) companion object { const val FIZZ_FIELD = "fizz" const val BIZZ_FIELD = "bizz" const val BAZZ_FIELD = "bazz" } } JSON-B A relative newcomer to the Java world—having been first released only in 2017 alongside JEE 8—JSON-B is an official standard for conducting serialization and deserialization for the JSON data format. The API uses either Eclipse Yasson or Apache Johnzon as the underlying implementation, meaning either one of these libraries would have to be included as a runtime dependency; the tests for this article used Yasson as the implementation. Usage The basic usage is to construct an instance of Jsonb via JsonbBuilder.create() and invoke the functions Jsonb.toJson() and Jsonb.fromJson() to serialize an object and deserialize a JSON string, respectively. Working With Kotlin JSON-B requires the most work of the four libraries evaluated to properly work with Kotlin. JSON-B serializes a class’s attributes in alphabetical order instead of declaration order. While this is not a deal-breaker—JSON objects do not require ordering for key fields—it is necessary to annotate a class with @JsonbPropertyOrder if specific ordering is desired: Kotlin @JsonbPropertyOrder("fizz", "bizz", "bazz") data class PojoFoo(var fizz: String, var bizz: Int, var bazz: List<String>) { constructor() : this("", 0, emptyList()) } Like Jackson, JSON-B requires a no-arg constructor and will fail if it does not encounter one while deserializing a JSON string into a class. Thus, a class without a no-arg constructor will need to mark the constructor that JSON-B needs to use with @JsonbCreator and mark each of the constructor’s arguments with @JsonbProperty so they correspond to the class’s attributes: Kotlin @JsonbPropertyOrder("fizz", "bizz", "bazz") data class ImmutableFoo @JsonbCreator constructor( @JsonbProperty("fizz") val fizz: String, @JsonbProperty("bizz") val bizz: Int, @JsonbProperty("bazz") val bazz: List<String> ) Lastly, JSON-B also shares Jackson’s trait of not being able to handle inline classes properly. Attempting to serialize ValueClassFoo will produce an incorrect output, and while JSON-B will not fail while trying to deserialize a string to ValueClassFoo, it will fail to populate the inline class attribute correctly: // Attempting to serialize to JSON expected: <{"fizz":"FUZZ","bizz":5,"bazz":["BUZZ","BOZZ"]}> but was: <{"bazz":["BUZZ","BOZZ"],"bizz-pVg5ArA":5,"fizz":"FUZZ"}> // Attempting to deserialize from JSON expected: <ValueClassFoo(fizz=FUZZ, bizz=5, bazz=[BUZZ, BOZZ])> but was: <ValueClassFoo(fizz=FUZZ, bizz=0, bazz=[BUZZ, BOZZ])> Like Jackson, the target class will need special serializer and deserializer classes to handle it and be annotated as such: Kotlin class ValueClassFooSerializer : JsonbSerializer<ValueClassFoo> { override fun serialize(valueClassFoo: ValueClassFoo, generator: JsonGenerator, ctx: SerializationContext?) { generator.writeStartObject() generator.write(ValueClassFoo.FIZZ_FIELD, valueClassFoo.fizz) generator.write(ValueClassFoo.BIZZ_FIELD, valueClassFoo.bizz.toInt()) generator.writeStartArray(ValueClassFoo.BAZZ_FIELD) valueClassFoo.bazz.forEach(generator::write) generator.writeEnd() generator.writeEnd() } } class ValueClassFooDeserializer : JsonbDeserializer<ValueClassFoo> { override fun deserialize(jsonParser: JsonParser, ctx: DeserializationContext?, rtType: Type?): ValueClassFoo { var fizz: String? = null var bizz: UInt? = null var bazz: List<String>? = null while (jsonParser.hasNext()) { val event = jsonParser.next() if (event != JsonParser.Event.KEY_NAME) continue when (jsonParser.string) { ValueClassFoo.FIZZ_FIELD -> { jsonParser.next() fizz = jsonParser.string } ValueClassFoo.BIZZ_FIELD -> { jsonParser.next() bizz = jsonParser.int.toUInt() } ValueClassFoo.BAZZ_FIELD -> { jsonParser.next() bazz = jsonParser.array.getValuesAs(JsonString::class.java).map { it.string } } } } if (fizz != null && bizz != null && bazz != null) { return ValueClassFoo(fizz = fizz, bizz = bizz, bazz = bazz) } else { throw IllegalStateException("'fizz', 'bizz', and 'bazz' must be not null") } } } @JsonbTypeDeserializer(ValueClassFooDeserializer::class) @JsonbTypeSerializer(ValueClassFooSerializer::class) data class ValueClassFoo(var fizz: String, var bizz: UInt, var bazz: List<String>) { constructor() : this("", 0u, emptyList()) companion object { const val FIZZ_FIELD = "fizz" const val BIZZ_FIELD = "bizz" const val BAZZ_FIELD = "bazz" } } Kotlinx Serialization Finally, the authors of Kotlin have published their own serialization library for the Kotlin programming language. First released in 2020, the Kotlinx Serialization library is designed for serialization actions in general, not just JSON; while the library only contains official support for JSON, it has experimental support for other formats like Protobuf and CBOR as well as community support for formats like YAML. Usage Unlike the other JSON serialization libraries, there is no instance object that needs to be created for conducting serialization actions. Instead, calls to the extension functions encodeToString() and decodeFromString() are made for the serializing object in question. In this case, the Kotlin object Json. Working With Kotlin Unlike the other JSON serialization libraries, Kotlinx Serialization does not work on custom classes by default. This is due to the way the library works: instead of using reflection like the other libraries, Kotlinx Serialization generates specific serialization and deserialization functions for the target class(es) at compile time. To recognize which classes need this serialization code generated for it, any target classes need to be annotated with @Serializable (a different method is available for third-party classes): Kotlin @Serializable data class PojoFoo(var fizz: String, var bizz: Int, var bazz: List<String>) { constructor() : this("", 0, emptyList()) } In addition, Kotlinx Serialization does not work by default on attributes with a default value. This needs to be enabled with the annotation @EncodeDefault: Kotlin @Serializable @OptIn(ExperimentalSerializationApi::class) data class DefaultValueFoo(@EncodeDefault val fizz: String = "FUZZ", var bizz: Int, var bazz: List<String>) { constructor() : this(bizz = 0, bazz = emptyList()) } Testing Parameters Each of the four JSON serialization libraries conducts serialization and deserialization of the four example classes, and the Java Microbenchmark Harness (JMH) benchmark tests measure the throughput of how many operations get executed per second on average. For example: Kotlin @State(Scope.Benchmark) open class SerializationComparison { private val gson = Gson() @Benchmark fun serializePojoFoo(): String = gson.toJson(pojoFoo) @Benchmark fun serializeImmutableFoo(): String = gson.toJson(immutableFoo) @Benchmark fun serializeDefaultValueFoo(): String = gson.toJson(defaultValueFoo) @Benchmark fun serializeValueClassFoo(): String = gson.toJson(valueClassFoo) @Benchmark fun deserializePojoFoo(): PojoFoo = gson.fromJson(pojoFooStr, PojoFoo::class.java) @Benchmark fun deserializeImmutableFoo(): ImmutableFoo = gson.fromJson(immutableFooStr, ImmutableFoo::class.java) @Benchmark fun deserializeDefaultValueFoo(): DefaultValueFoo = gson.fromJson(defaultValueFooStr, DefaultValueFoo::class.java) @Benchmark fun deserializeValueClassFoo(): ValueClassFoo = gson.fromJson(valueClassFooStr, ValueClassFoo::class.java) } These tests utilize JMH’s defaults of: Five warmup rounds of ten seconds. Five rounds of measurements. Five forked processes to conduct both of the above. The tests are run on a macOS with an Intel Core i7 2.6 GHz 6-Core and 16GB of RAM; the executing JVM is Temurin 19+36. Results Serialization The clear winner among the four libraries is Kotlinx Serialization, as it averages over 5 million operations per second, much faster than the second-place Jackson library. It’d be impossible to identify the exact reasons for why the performance of Kotlinx Serialization is so much higher compared to the competition without diving too deeply into the source code of each library, but a hint may lie in how the other libraries perform much better during the serialization of ValueClassFoo compared to the other example classes (the exception is Kotlinx Serialization, which appears to do worse, but given the error ranges for each result, it’s not statistically significant). For example, running the Java Flight Recorder profiler on Jackson provides the following result in the call tree for serializing PojoFoo: In contrast, here is the call tree for serializing ValueClassFoo: As the two call trees show, creating a special class for the serialization of instances of ValueClassFoo means Jackson does not have to use reflection—a very expensive process, computationally-speaking—to determine what attributes need to be serialized. Of course, this comes with the downside of having more code for the developer to maintain, and will break as soon as the class’s attributes are modified. Deserialization Again, Kotlinx Serialization clearly performs better for deserializing compared to the remaining three libraries. GSON, Jackson, and Kotlinx Serialization all performed markedly better when deserializing instances of DefaultValueFoo, and that’s presumably because there were fewer data to read in for the deserialization test—for that scenario, the libraries had to deserialize {"bizz":5,"bazz":["BUZZ","BOZZ"]}, meaning one less field to parse. Interestingly, Jackson did worse in deserializing ValueClassFoo compared to the other example classes. Again using the Java Flight Recorder profiler, here is a flame graph for Jackson deserializing PojoFoo: Likewise, here is a flame graph for Jackson deserializing ValueClassFoo: It appears that, in contrast to serialization actions, Jackson’s default deserializer is faster than a hand-rolled deserializer. Of course, there wasn’t a choice for doing this in the case of an inline class: it was either creating the custom deserializer or having the code crash. Final Thoughts While the tests provide promising results for the Kotlinx Serialization library, there are a few caveats that must be provided: The example classes were relatively simple to reduce the amount of variables between testing scenarios. Conducting serialization and deserialization actions on large and complex data structures might provide entirely different results in favor of a different serialization library. Due to the Kotlinx Serialization code being developed for the Kotlin programming language, code written in Java would have to be rewritten in Kotlin to use the library, something that might be a very time-consuming endeavor and a hard sell for a project that has a large code base written in Java. The other three libraries, on the other hand, have no such restriction and can be used with Java and Kotlin alike. Regardless, the results suggest that it would behoove Kotlin developers to give the Kotlinx Serialization library a try in their projects, as aside from the high performance, it also provides the opportunity to be a “one-stop shop” for serialization not only for JSON but for other formats like Protobuf, YAML, and more.

I recently came across this blog post from Ruud van Asseldonk titled “The YAML Document From Hell.” I’ve always heard that YAML has its pitfalls, but hadn’t looked into the details and thankfully hadn’t been affected, mainly due to my very infrequent and simple use of YAML. If you are in the same boat as me, I recommend reading that article now, as I almost can’t believe I’ve avoided any issues with it. The article digs into the issues in the YAML spec itself and then describes what happens in Python’s PyYAML and Golang’s YAML library with an example file, the titular YAML document from hell. I wanted to see how things were in the JavaScript ecosystem. YAML in JavaScript A search for JavaScript YAML parsers on npm brings up YAML (which I have used in my own project) and js-yaml. js-yaml has the most weekly downloads according to npm and the most stars on GitHub however, YAML seems to be under more active development, having been most recently published (a month ago at the time of writing) compared to js-yaml’s last publish date almost 2 years ago. There is also yamljs, but the project hasn’t received a commitment since November 2019 and hasn’t been released for 6 years, so I am going to disregard it for now. Let’s see what YAML and js-yaml do with the YAML document from hell. The Document Itself To save yourself from going back and forth between van Asseldonk’s article and this one, here is the YAML document. server_config: port_mapping: # Expose only ssh and http to the public internet. - 22:22 - 80:80 - 443:443 serve: - /robots.txt - /favicon.ico - *.html - *.png - !.git # Do not expose our Git repository to the entire world. geoblock_regions: # The legal team has not approved distribution in the Nordics yet. - dk - fi - is - no - se flush_cache: on: [push, memory_pressure] priority: background allow_postgres_versions: - 9.5.25 - 9.6.24 - 10.23 - 12.13 So how do our JavaScript libraries handle this file? The Failures Anchors, Aliases, and Tags Let’s start with the failures. As described in the original article under the subhead “Anchors, aliases, and tags” this section is invalid: serve: - /robots.txt - /favicon.ico - *.html - *.png - !.git # Do not expose our Git repository to the entire world. This causes both of our JavaScript YAML libraries to throw an error, both referencing an undefined alias. This is because the * is a way to reference an anchor created earlier in the document using an &. In our document’s case, that anchor was never created, so this is a parsing error. If you want to learn more about anchors and aliases it seems like something that is important in build pipelines. Both Bitbucket and GitLab have written about how to use anchors to avoid repeating sections in yaml files. For the purposes of trying to get the file to parse, we can make those aliases strings as they were likely intended. serve: - /robots.txt - /favicon.ico - "*.html" - "*.png" - !.git # Do not expose our Git repository to the entire world. Now we get another parsing error from our libraries; both of them complain about an unknown or unresolved tag. The ! at the start of !.git is the character triggering this behaviour. Tags seem to be the most complicated part of YAML to me. They depend on the parser you are using and allow that parser to do something custom with the content that follows the tag. My understanding is that you could use this in JavaScript to, say, tag some content to be parsed into a Map instead of an Object or a Set instead of an Array. Van Asseldonk explains this with this alarming sentence: This means that loading an untrusted YAML document is generally unsafe, as it may lead to arbitrary code execution. PyYaml apparently has a safe_load method that will avoid this, but Go’s yaml package doesn’t. It seems that the JavaScript libraries also lack this feature, so the warning for untrusted YAML documents stands. If you do want to take advantage of the tag feature in yaml, you can check out the yaml package’s documentation on custom data types or js-yaml’s supported yaml types and unsafe type extensions. To make the YAML file parse, let’s encase all the weird yaml artifacts in quotes to make them strings: serve: - /robots.txt - /favicon.ico - "*.html" - "*.png" - "!.git" # Do not expose our Git repository to the entire world. With the serve block looking it does above, the file now parses. So what happens to the rest of the potential yaml gotchas? Accidental Numbers One thing that I am gathering from this investigation so far is that if you need something to be a string, do not be ambiguous about it, surround it in quotes. That counted for the aliases and tags above and it also counts for accidental numbers. In the following section of the yaml file you see a list of version numbers: allow_postgres_versions: - 9.5.25 - 9.6.24 - 10.23 - 12.13 Version numbers are strings, numbers can’t have more than one decimal point in them. But when this is parsed by either JavaScript library the result is as follows: allow_postgres_versions: [ '9.5.25', '9.6.24', 10.23, 12.13 ] Now we have an array of strings and numbers. If a YAML parser thinks something looks like a number it will parse it as such. And when you come to use those values they might not act as you expect. Version Numbers in GitHub Actions I have had this issue within GitHub Actions before. It was in a Ruby project, but this applies to anyone trying to use version numbers in a GitHub Actions YAML file. I tried to use a list of Ruby version numbers, this worked fine up until Ruby version 3.1 was released. I had 3.0 in the array. Within GitHub Actions this was parsed as the integer 3. This might seem fine, except that when you give an integer version to GitHub Actions it picks the latest minor point for that version. So, once Ruby 3.1 was released, the number 3.0 would select version 3.1. I had to make the version number a string, "3.0", and then it was applied correctly. Accidental numbers cause issues. If you need a string, make sure you provide a string. The Successes It’s not all bad in the JavaScript world. After working through the issues above, we might now be in the clear. Let’s take a look now at what parsed correctly from this YAML file. Sexagesimal Numbers Under the port mapping section of the YAML file we see: port_mapping: # Expose only ssh and http to the public internet. - 22:22 - 80:80 - 443:443 That 22:22 is dangerous in yaml version 1.1 and PyYaml parses it as a sexagesimal (base 60) number, giving the result of 1342. Thankfully both JavaScript libraries have implemented YAML 1.2 and 22:22 is parsed correctly as a string in this case. port_mapping: [ '22:22', '80:80', '443:443' ] The Norway Problem In YAML 1.1 no is parsed as false. This is known as “the Norway problem” because listing countries as two character identifiers is fairly common and having this YAML: geoblock_regions: - dk - fi - is - no - se Parsed into this JavaScript: geoblock_regions: [ 'dk', 'fi', 'is', false, 'se' ] It is just not helpful. The good news is that, unlike Go’s YAML library, both JavaScript libraries have implemented YAML 1.2 and dropped no as an alternative for false. The geoblock_regions sections is successfully parsed as follows: geoblock_regions: [ 'dk', 'fi', 'is', 'no', 'se' ] Non-String Keys You might believe that keys in YAML would be parsed as strings, like JSON. However they can be any value. Once again there are values that may trip you up. Much like with the Norway problem in which yes and no can be parsed as true and false, the same goes for on and off. This is manifested in our YAML file in the flush_cache section: flush_cache: on: [push, memory_pressure] priority: background Here the key is on, but in some libraries it is parsed as a boolean. In Python, even more confusingly the boolean is then stringified and appears as the key "True". Thankfully this is handled by the JavaScript libraries and on becomes the key "on". flush_cache: { on: [ 'push', 'memory_pressure' ], priority: 'background' } This is of particular concern in GitHub Actions again, where on is used to determine what events should trigger an Action. I wonder if GitHub had to work around this when implementing their parsing. Parsing as YAML Version 1.1 Many of the issues that our JavaScript libraries sidestep are problems from YAML 1.1 and both libraries have fully implemented YAML 1.2. If you do wish to throw caution to the wind, or you have to parse a yaml file explicitly with YAML 1.1 settings, the YAML library can do that for you. You can pass a second argument to the parse function to tell it to use version 1.1, like so: import { parse } from "yaml"; const yaml = parse(yamlContents, { version: "1.1" }); console.log(yaml); Now you get a result with all of the fun described above: { server_config: { port_mapping: [ 1342, '80:80', '443:443' ], serve: [ '/robots.txt', '/favicon.ico', '*.html', '*.png', '!.git' ], geoblock_regions: [ 'dk', 'fi', 'is', false, 'se' ], flush_cache: { true: [ 'push', 'memory_pressure' ], priority: 'background' }, allow_postgres_versions: [ '9.5.25', '9.6.24', 10.23, 12.13 ] } } Note that in this case I left the aliases and tags quoted as strings so that the file could be parsed successfully. Stick with version 1.2, the default in both JavaScript YAML libraries, and you’ll get a much more sensible result. Isn’t YAML Fun? In this post we’ve seen that it’s easy to write malformed YAML if you weren’t aware of aliases or tags. It’s also easy to write mixed arrays of strings and numbers. There are also languages and libraries in which YAML 1.1 is still hanging around and on. yes, off, and no are booleans and some numbers can be parsed into base 60. My advice, after going through all of this, is to err on the side of caution when writing YAML. If you want a key or a value to be a string, surround it in quotes and explicitly make it a string. On the other hand, if you are parsing someone else’s yaml then you will need to program defensively and try to handle the edge cases, like accidental numbers, that can still cause issues. Finally, if you have the option, choose a different format to YAML. YAML is supposed to be human-friendly, but the surprises and the bugs that it can produce are certainly not developer-friendly and ultimately that defeats the purpose. The conclusion to the original YAML document from hell post suggests many alternatives to YAML that will work better. I can’t help but think that in the world of JavaScript that something JSON based, but friendlier to author, should be the solution. There is a package that simply strips comments from JSON or there’s JSON5 a JSON format that aims to be easier to write and maintain by hand. JSON5 supports comments as well as trailing commas, multiline strings, and various number formats. Either of these are a good start if you want to make authoring JSON easier and parsing hand authored files more consistent. If you can avoid YAML, I recommend it. If you can’t, good luck.

Regular expressions are effective tools for pattern matching and text processing. A regex, or regular expression, is a group of characters that forms a search pattern. To determine whether a string contains a particular search pattern, use RegEx. They are supported by many programming languages, including Python, which provides a powerful and flexible regular expression engine that can handle a wide range of text-matching tasks. This article will provide a brief guide to using regular expressions in Python. What Are Regular Expressions? Regular Expressions (RegEx) are unique character combinations that use a search pattern to locate a string or group of strings, such as finding all email addresses in a document or validating the format of a phone number. It can distinguish between the presence and absence of a text by comparing it to a specific pattern. It may also divide a pattern into one or more sub-patterns. The use of regex in Python is supported via the re-module, which is provided by Python. Its main purpose is to provide a search; to do this, a regular expression and a string are required. In this case, it either returns the first match or none at all. Regular expressions are used to match patterns in text. Regular expressions are often used in text editors, command-line utilities, and programming languages. Regular expressions consist of two types of characters: Literals: These are characters that match themselves. For example, the letter "a" will match the letter "a" in a text string. Metacharacters: These are special characters that have a special meaning. For example, the dot (.) metacharacter matches any single character. Using Regular Expressions in Python Python provides a built-in module called "re" that provides regular expression support. This module provides several functions for working with regular expressions, including searching for matches, replacing matches, and splitting a string into a list of substrings based on a pattern. The "re" module also provides several special characters that can be used to create complex regular expressions. Here are some of the most commonly used special characters in regular expressions: Character Description . any single character, excluding the newline (\n), is matched. For example, the regular expression for he.. will match for “hell”,” help,” etc. * compares to 0 or more occurrences of the preceding character. For example, the regular expression a* will match zero or more occurrences of the letter "a". + matches to one or more occurrences of the preceding character. For example, the regular expression a+ will match one or more occurrences of the letter "a". ? matches either zero or one instance of the preceding character. For example, the regular expression colo?r will match both "color" and "colour". {m,n} Matches the previous character between m and n times. For example, the regular expression a{2,3} will match either "aa" or "aaa". [] Matches any single character within the brackets. For instance, the regular expression [aeiou] will match any kind of vowel. \ Used to drop the special meaning of the character following it. For example, the regular expression \. will match a period character. ^ The string should start with the characters following ^. For example, the regular expression ^hello will match only if the sentence starts with hello. $ The string should end with the characters following $. For example, the regular expression hello$ will match only if the sentence ends with hello. | Either or. For example, the regular expression suman|ritik Check if the string contains either "suman" or "ritik" Let’s discuss some important of these metacharacters in detail: . – Dot Except for the newline character (\n), the dot (.) symbol only recognizes one character. For instance: a.b will look for any character other than a dot in the string, including acb, acbd, abbb, etc. .. will determine whether the string has at least two characters. * – Star Star (*) symbol matches zero or more instances of the regex that comes before the star symbol. For instance: Because b is not followed by c, ab*c will be matched for the strings ac, abc, abbbc, dabc, etc., but not for abdc. + - Plus One or more instances of the regex that comes before the + symbol are matched by the Plus (+) symbol. For instance : Because there is no b in ac and b is not followed by c in abdc, ab+c will match for the string abc, abbc, dabc but not for ac, abdc. ? - Question The question mark (?) determines whether the string in the regex appears at least once or not at all. For instance: As there are two b’s in the string abbc, it will not be matched. However, ab?c will be matched for the strings ac, acb, and dabc. Because b is not followed by c, it will also not match for abdc. Braces {m, n} All repetitions from m to n, inclusive, before the regex are matched by the braces. Example- The strings aaab, baaac, and gaad will be matched for regular expression a{2, 4}, but it won't be matched for strings like abc, bc because there is either just one an or none at all in both situations. Square brackets []. A character class made up of a group of characters that we want to match is represented by square brackets ([]). The character class [abc] will, for instance, match any single a, b, or c. With the - symbol between the square brackets, we can also specify a range of characters. For instance: The sample for [0123] is [0,3]. The sample for [abc] is [a-c]. The caret(^) sign can be used to reverse the character class as well. For instance: [^0-3] denotes any number other than 0 and 1 or 3. [^a-c]Any character that is not an a, b, or c. \ Backslash To ensure that the character is not given special treatment, use the backslash (/). This could be thought of as a metacharacter escape. As an illustration, the dot (.) will be treated as a special character and one of the metacharacters if you want to search for it in the string (as shown in the above table). In order to prevent it from losing its specialization, we will employ the backslash (/) before the dot (.) in this instance. The example below will help you understand. Code: Python import re s = 'suman.singh' # without using \ match = re.search(r'.', s) print(match) # using \ match = re.search(r'\.', s) print(match) Output: Python <re.Match object; span=(0, 1), match='s'> <re.Match object; span=(6, 7), match='.'> | - Or Symbol Determines whether the pattern before or after the or symbol is present in the string. For instance: Any string that contains either an or b, such as acd, bcd, abcd, etc., will be matched by a|b. Special Sequences Special sequences provide the precise position in the search string where the match must take place rather than matching for the actual character in the string. It makes it simpler to write patterns that are used frequently. Special Sequences List Special Sequence Description Examples \A matches if the specified character appears at the start of the string. \Afor -> for suman \b Matches if the provided character either starts or finishes the word. \b(string) will look for the word's beginning, and \b(string) will look for the word's ending. \bsh -> suman \B In contrast to the \b, the string shall not begin or end with the specified pattern. \Bge -> together \d This is similar to the set class [0-9] because it matches any decimal digit. \d -> 1526 \D matches any character that is not a digit; this is the same as the set class [0-9]. \D -> suman \s each whitespace character is a match. \s -> sum an \S any non-whitespace character is a match. \S -> s uman \w This is comparable to the class [a-zA-Z0-9_] and matches any alphanumeric character. \w -> 3425 \W any non-alphanumeric character is matched. \W -> >$ \Z matches if the string contains the specified regex at the end. an\Z -> suman Basic Regular Expression Operations 1. Searching for Matches The most basic operation in regular expressions is searching for a match in a string. The "re" module provides the "search" function for this purpose. Here is an example of how to use the "search" function to find a pattern in a string: Code: Python import re text = "Suman Raghav and Ron are friends" pattern = "friends" result = re.search(pattern, text) if result: print("String Pattern Found") else: print("String Pattern not Found") This code will output "String Pattern Found" because the pattern "friends" is found in the text. 2. Replacing Matches Another common operation in regular expressions is replacing matches in a string. The "re" module provides the "sub" function for this purpose. Here is an example of how to use the "sub" function to replace a pattern in a string: Code: Python import re text = "Suman Raghav and Ron are friends" pattern = "friends" replacement = "students" result = re.sub(pattern, replacement, text) print(result) This code will output "Suman Raghav and Ron are students" because the pattern "friends" is replaced with "students" in the original text. 3. Splitting a String Based on a Pattern The "re" module can also be used to split a string into a list of substrings based on a pattern. The split function is used for this purpose. Here is an example of how to use the "split" function to split a string based on whitespace characters: Code: Python import re text = "Suman Raghav and Ron are friends" result = re.split("\s", text) print(result) This code will output ["Suman", "Raghav", "and", "Ron", "are", "friends"] because the string is split based on whitespace characters. 4. Regular Expression Flags Regular expressions in Python support flags that modify the behavior of the regular expression engine. Flags are specified as an optional second argument to the regular expression function. Some of the most widely used flags are listed below: re.IGNORECASE or re.I: Makes the regular expression case-insensitive. re.MULTILINE or re.M: Allows the ^ and $ metacharacters to match the beginning and end of each line in a multiline string rather than just the beginning and end of the entire string. re.DOTALL or re.S: Makes the dot (.) metacharacter match any character, including a newline character (\n). re.ASCII or re.A: Limits the regular expression engine to ASCII characters only. Here is an example of how to use the IGNORECASE flag to make a regular expression case-insensitive: Code: Python import re text = "Suman has a brown coloured bag." pattern = "BROWN" result = re.search(pattern, text, re.IGNORECASE) if result: print("String Pattern Found") else: print("String Pattern not Found") This code will output "String Pattern Found" because the pattern "BROWN" is found in the text, even though it is in uppercase and the search was performed with the IGNORECASE flag. 5. Grouping and Capturing Regular expressions in Python also support the grouping and capturing of substrings within a match. Grouping is achieved using parentheses (()). The contents of the first group are captured and can be accessed using the "group" method of the match object. Here is an example of how to use grouping and capturing in regular expressions: Code: Python import re text = "Suman Singh (sumansingh@example.com) wrote an email" pattern = "(\w+@\w+\.\w+)" result = re.search(pattern, text) if result: print("Email address validated: " + result.group(1)) else: print("Email address not validated") This code will output "Email address validated: sumansingh@example.com" because the regular expression matches the email address in the text and captures it using a group. Conclusion For text processing and pattern matching, regular expressions are an effective tool. The "re" module in Python provides a flexible and powerful regular expression engine. Special characters such as ., *, +, ?, ^, $, [], (), and | are used to define patterns in regular expressions. The most commonly used regular expression functions in Python are "search", "match", "findall", "sub", and "split". Regular expression flags such as re.IGNORECASE, re.MULTILINE, re.DOTALL, and re.ASCII can modify the behavior of the regular expression engine. Grouping and capturing of substrings within a match can be achieved using parentheses (()) and the "group" method of the match object.

The real name of the CSS variables is CSS Custom Properties is a draft standard (yes, when I wrote these lines, it is on Candidate Recommendation Snapshot), but it is widely supported by modern browsers. CSS variables allow us, like another kind of variable in another programming language, to store a value we can reuse across our document. For example, if we define a CSS variable for the primary color doing the following: --primary-color: #f00;, then we can use it in any component like: .my-component { color: var(--primary-color); } Usually, you “attach” your variable to :root, which means the variable will be available in all the document :root { color: var(--primary-color); } In this example :root is the variable scope. Using Together SCSS If you want to assign values from SCSS variables to CSS variables, you can not do the “normal” notation: // ❌ This doesn't work $scss-var: #f00; --my-var: $scss-var; In the example, the value of --my-var is literally $scss-var, not the value of $scss-var, this behavior was done to provide maximum compatibility with the plain CSS. To make it work, you need to use the Sass interpolation syntax: #{my scss script code}: // ✅ This works $scss-var: #f00; --my-var: #{$scss-var}; Scope The variables are only available in the element where it is defined and its children; that is the scope of the variable. Outside there, the variable doesn’t exist. If you try to access to use a variable that is not in the scope, you will not get an error, but the property that is using the not existing variable will be ignored. Hoisting Like the JS variables, the CSS variables are moved to the top, so you can use them before defining them. .my-element { color: var(--primary-color); } :root { --primary-color: #f00; } Override As I mentioned before, the variables have a scope where the variable exists, but: what happens if a variable with the same name is defined in two scopes: It happens the same as in a JS variable; the near local scope overrides other values: :root { --color: #0f0; } .my-element { --color: #0ff; color: var(--color); } This behavior is very convenient when we work with UI components with different styles depending on modifiers. CSS Variables in UI components Imagine we have a simple button component like that. <button class="ui-button"> Button content </button> .ui-button { background: #333; color: #fff; font-size: 12px; padding: 4px 10px; } This button has different variants by color (default, red and green) and size (default, small and big); using BEM, we can add a modifier class like .ui-button--green or .ui-button--big and use that to overwrite the styles, for example: .ui-button { background: #333; color: #fff; font-size: 12px; padding: 4px 10px; &--green { background: #1F715F; } &--big { font-size: 16px; padding: 6px 20px; } } This way works perfectly, but we need to know which properties to overwrite, and need to do it explicitly for each modifier, so it’s easy to forget something, or if we need to add a new property affected by the modifiers, add it in all of them Suppose we rewrite the styles using CSS variables, parameterizing the component styles. In that case, we can override the CSS variable values for each modifier without changing the CSS styles itself for the modifiers, only changing the value of the variables: .ui-button { --bg-color: #333; --text-color: #fff; --font-size: 12px; --padding: 4px 10px; background: var(--bg-color); color: var(--text-color); font-size: var(--font-size); padding: var(--padding); &--green { --bg-color: #1F715F; } &--red { --bg-color: #0ff; } &--big { --font-size: 16px; --padding: 6px 20px; } &--small { --font-size: 10px; --padding: 3px 5px; } } Variable Scope Priority In CSS, the elements can use more than a class, so that means the element’s CSS variables have multiple scopes at the same level; for example, if we apply the green and red modifiers at the same time <button class="ui-button ui-button--green ui-button--red"> Green + red </button> Both ui-button--green and ui-button--red define the same --bg-color variable, What value will be applied to the element? In cases like that, the class order is the priority, so the last class used overrides the value last, and its value is applied; in the example, the button will be red, but for <button class="ui-button ui-button--red ui-button--green"> the button will be green. Summarizing The use of CSS variables and scopes is a powerful tool when you are developing components in general. Still, if your components have modifiers, it requires extra work in the beginning to parameterize the component, but after that makes it simpler to create variants and modifiers.

We will go over Apache Kafka basics, installation, and operation, as well as a step-by-step implementation using a .NET Core 6 web application. Prerequisites Visual Studio 2022 .NET Core 6 SDK SQL Server Java JDK 11 Apache Kafka Agenda Overview of Event Streaming Introduction to Apache Kafka. Main concepts and foundation of Kafka. Different Kafka APIs. Use cases of Apache Kafka. Installation of Kafka on Windows 10. Step-by-step implementation Overview of Event Streaming Events are the things that happen within our application when we navigate something. For example, we sign up on any website and order something, so, these are the events. The event streaming platform records different types of data like transaction, historical, and real-time data. This platform is also used to process events and allow different consumers to process results immediately and in a timely manner. An event-driven platform allows us to monitor our business and real-time data from different types of devices like IoT and many more. After analyzing, it provides a good customer experience based on different types of events and needs. Introduction to Apache Kafka Below, are a few bullet points that describe Apache Kafka: Kafka is a distributed event store and stream-processing platform. Kafka is open source and is written in Java and Scala. The primary purpose to designed Kafka by Apache foundation is to handle real-time data feeds and provide high throughput and low latency platforms. Kafka is an event streaming platform that has many capabilities to publish (write) and subscribe to (read) streams of events from a different system. Also, to store and process events durably as long as we want, by default, Kafka stores events from seven days of the time period, but we can increase that as per need and requirement. Kafka has distributed system, which has servers and clients that can communicate via TCP protocol. It can be deployed on different virtual machines and containers in on-premise and cloud environments as per requirements. In the Kafka world, a producer sends messages to the Kafka broker. The messages will get stored inside the topics and the consumer subscribes to that topic to consume messages sent by the producer. ZooKeeper is used to manage the metadata of Kafka-related things, it tracks which brokers are part of the Kafka cluster and partitions of different topics. Lastly, it manages the status of Kafka nodes and maintains a list of Kafka topics and messages. Main Concepts and Foundation of Kafka 1. Event An event or record is the message that we read and write to the Kafka server; we do this in the form of events in our business world, and it contains a key, a value, a timestamp, and other metadata headers. The key, value, and time stamp, in this case, are as follows: Key: “Jaydeep” Value: “Booked BMW” Event Timestamp: “Dec. 11, 2022, at 12:00 p.m.” 2. Producer The producer is a client application that sends messages to the Kafka node or broker. 3. Consumer The consumer is an application that receives data from Kafka. 4. Kafka Cluster The Kafka cluster is the set of computers that share the workload with each other with varying purposes. 5. Broker The broker is a Kafka server that acts as an agent between the producer and consumer, who communicate via the broker. 6. Topic The events are stored inside the “topic,” it’s similar to our folder in which we store multiple files. Each topic has one or more producers and consumers, which write and reads data from the topic. Events in “topic” can be read as often as needed because it persists events and it’s not like another messaging system that removes messages after consuming. 7. Partitions Topics are partitions, meaning the topic is spread over multiple partitions that we created inside the topic. When the producer sends some event to the topic, it will store it inside the particular partitions, and then, the consumer can read the event from the corresponding topic partition in sequence. 8. Offset Kafka assigns one unique ID to the message stored inside the topic partition when the message arrives from the producer. 9. Consumer Groups In the Kafka world, the consumer group acts as a single logical unit. 10. Replica In Kafka, to make data fault-tolerant and highly available, we can replicate topics in different regions and brokers. So, in case something wrong happens with data in one topic, we can easily get that from another to replicate the same. Different Kafka APIs Kafka has five core APIs that serve different purposes: Admin API: This API manages different topics, brokers, and Kafka objects. Producer API: This API is used to write/publish events to different Kafka topics. Consumer API: This API is used to receive the different messages corresponding to the topics that are subscribed by the consumer. Kafka Stream API: This API is used to perform different types of operations like windowing, joins, aggregation, and many others. Basically, its use is to transform objects. Kafka Connect API: This API works as a connector to Kafka, which helps different systems connect with Kafka easily. It has different types of ready-to-use connectors related to Kafka. Use Cases of Apache Kafka Messaging User activity tracking Log aggregation Stream processing Realtime data analytics Installation of Kafka on Windows 10 Step 1 Download and install the Java SDK of version 8 or more. Note: I have Java 11, that’s why I put the same path in all commands that I used here. Step 2 Open and install EXE. Step 3 Set the environment variable for Java using the command prompt as admin. Command: setx -m JAVA_HOME “C:\Program Files\Java\jdk-11.0.16.1” setx -m PATH “%JAVA_HOME%\bin;%PATH%” Step 4 After that, download and install Apache Kafka. Step 5 Extract the downloaded Kafka file and rename it “Kafka.” Step 6 Open D:\Kafka\config\ and create a “zookeeper-data” and “kafka-logs” folder inside that. Step 7 Next, open D:\Kafka\config\zookeeper.properties file and add the folder path inside that: D:\Kafka\config\zookeeper.properties dataDir=D:/Kafka/zookeeper-data Step 8 After that, open D:\Kafka\config\server.properties file and change the log path over there: D:\Kafka\config\server.properties log.dirs=D:/Kafka/kafka-logs Step 9 Saves and close both files. Step 10 Run ZooKeeper: D:\Kafka> .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties Step 11 Start Kafka: D:\Kafka> .\bin\windows\kafka-server-start.bat .\config\server.properties Step 12 Create Kafka topic: D:\Kafka\bin\windows>kafka-topics.bat — create — bootstrap-server localhost:9092 — replication-factor 1 — partitions 1 — topic testdata Step 13 Create a producer and send some messages after you’ve started a producer and consumer: D:\Kafka\bin\windows>kafka-console-producer.bat — broker-list localhost:9092 — topic testdata Step 14 Next, create a consumer. After, you will see the message the producer sent: D:\Kafka\bin\windows>kafka-console-consumer.bat — bootstrap-server localhost:9092 — topic testdata Step-by-Step Implementation Let’s start with practical implementation. Step 1 Create a new .NET Core Producer Web API: Step 2 Configure your application: Step 3 Provide additional details: Step 4 Install the following two NuGet packages: Step 5 Add configuration details inside the appsettings.json file: JSON { "Logging": { "LogLevel": { "Default": "Information", "Microsoft.AspNetCore": "Warning" } }, "AllowedHosts": "*", "producerconfiguration": { "bootstrapservers": "localhost:9092" }, "TopicName": "testdata" } Step 6 Register a few services inside the “Program” class: C# using Confluent.Kafka; var builder = WebApplication.CreateBuilder(args); // Add services to the container. var producerConfiguration = new ProducerConfig(); builder.Configuration.Bind("producerconfiguration", producerConfiguration); builder.Services.AddSingleton<ProducerConfig>(producerConfiguration); builder.Services.AddControllers(); // Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle builder.Services.AddEndpointsApiExplorer(); builder.Services.AddSwaggerGen(); var app = builder.Build(); // Configure the HTTP request pipeline. if (app.Environment.IsDevelopment()) { app.UseSwagger(); app.UseSwaggerUI(); } app.UseHttpsRedirection(); app.UseAuthorization(); app.MapControllers(); app.Run(); Step 7 Next, create the CarDetails model class: C# using Microsoft.AspNetCore.Authentication; namespace ProducerApplication.Models { public class CarDetails { public int CarId { get; set; } public string CarName { get; set; } public string BookingStatus { get; set; } } } Step 8 Now, create the CarsController class: C# using Confluent.Kafka; using Microsoft.AspNetCore.Mvc; using Microsoft.Extensions.Configuration; using Newtonsoft.Json; using ProducerApplication.Models; namespace ProducerApplication.Controllers { [Route("api/[controller]")] [ApiController] public class CarsController : ControllerBase { private ProducerConfig _configuration; private readonly IConfiguration _config; public CarsController(ProducerConfig configuration, IConfiguration config) { _configuration = configuration; _config = config; } [HttpPost("sendBookingDetails")] public async Task<ActionResult> Get([FromBody] CarDetails employee) { string serializedData = JsonConvert.SerializeObject(employee); var topic = _config.GetSection("TopicName").Value; using (var producer = new ProducerBuilder<Null, string>(_configuration).Build()) { await producer.ProduceAsync(topic, new Message<Null, string> { Value = serializedData }); producer.Flush(TimeSpan.FromSeconds(10)); return Ok(true); } } } } Step 9 Finally, run the application and send a message: Step 10 Now, create a “consumer” application: For that, create a new .NET Core console application: Step 11 Configure your application: Step 12 Provide additional information: Step 13 Install the NuGet below: Step 14 Add the following code, which consumes messages sent by the consumer: C# using Confluent.Kafka; var config = new ConsumerConfig { GroupId = "gid-consumers", BootstrapServers = "localhost:9092" }; using (var consumer = new ConsumerBuilder<Null, string>(config).Build()) { consumer.Subscribe("testdata"); while (true) { var bookingDetails = consumer.Consume(); Console.WriteLine(bookingDetails.Message.Value); } } Step 15 Finally, run the producer and consumer, send a message using the producer app, and you will see the message immediately inside the consumer console sent by the producer: Here is the GitHub URL I used in this article. Conclusion Here, we discussed Apache Kafka introduction, working, benefits, and step-by-step implementation using .NET Core 6. Happy Coding!

Retrieving the page source of a website under scrutiny is a day-to-day task for most test automation engineers. Analysis of the page source helps eliminate bugs identified during regular website testing, functional testing, or security testing drills. In an extensively complex application testing process, automation test scripts can be written in a way that if errors are detected in the program, then it automatically: Saves that particular page’s source code. Notifies the person responsible for the URL of the page. Extracts the HTML source of a specific element or code-block and delegates it to the responsible authorities if the error has occurred in one particular independent HTML WebElement or code block. This is an easy way to trace and fix logical and syntactical errors in the front-end code. In this article, we first understand the terminologies involved and explore how to get the page source in Selenium WebDriver using Python. What Is an HTML Page Source? In non-technical terminology, it’s a set of instructions for browsers to display info on the screen in an aesthetic fashion. Browsers interpret these instructions in their own ways to create browser screens for the client-side. These are usually written using HyperText Markup Language (HTML), Cascading Style Sheets (CSS), and Javascript. This entire set of HTML instructions that make a web page is called page source, HTML source, or simply source code. Website source code is a collection of source code from individual web pages. Here’s an example of a source code for a basic page with a title, form, image, and submit button. <!DOCTYPE html> <html> <head> <title>Page Source Example - LambdaTest</title> </head> <body> <h2>Debug selenium testing results : LambdaTest</h2> <img loading="lazy" data-fr-src="https://cdn.lambdatest.com/assetsnew/images/debug-selenium-testing-results.jpg" alt="debug selenium testing" width="550" height="500"><br><br> <form action="/"> <label for="debug">Do you debug test results using LambdaTest?</label><br> <input type="text" id="debug" name="debug" value="Of-course!"><br> <br> <input type="submit" value="Submit"> </form> <br><br> <button type="button" onclick="alert('Page Source Example : LambdaTest!')">Click Me!</button> </body> </html> What Is an HTML Web Element? The easiest way to describe an HTML web element would be, “any HTML tag that constitutes the HTML page source code is a web element.” It could be an HTML code block, an independent HTML tag like </br>, a media object on the web page—image, audio, video, a JS function, or a JSON object wrapped within <script> </script> tags. In the above example, <title> is an HTML web element, and the children of body tags are HTML web elements too, i.e., <img>, <button>, etc. How To Get Page Source in Selenium WebDriver Using Python Selenium WebDriver is a robust automation testing tool and provides automation test engineers with a diverse set of ready-to-use APIs. To make Selenium WebDriver get page source, Selenium Python bindings provide us with a driver function called page_source to get the HTML source of the currently active URL in the browser. Alternatively, we can also use the GET function of Python’s request library to load the page source. Another way is to execute JavaScript using the driver function execute_script and make Selenium WebDriver get page source in Python. A unrecommended way of getting page source is using XPath in tandem with the “view-source:” URL. Let’s explore examples for these four ways of how to get page source in Selenium WebDriver using Python. We’ll be using a sample small web page hosted on GitHub for all four examples. This page was created to demonstrate drag and drop testing in Selenium Python using LambdaTest. Get HTML Page Source Using driver.page_source We’ll fetch pynishant.github.io in the ChromeDriver and save its content to a file named page_source.html. This file name could be anything of your choice. Next, we read the file’s content and print it on the terminal before closing the driver: from selenium import webdriver driver = webdriver.Chrome() driver.maximize_window() driver.get("https://pynishant.github.io/") pageSource = driver.page_source fileToWrite = open("page_source.html", "w") fileToWrite.write(pageSource) fileToWrite.close() fileToRead = open("page_source.html", "r") print(fileToRead.read()) fileToRead.close() driver.quit() On successful execution of the above script, your terminal output will show the following page source: Get HTML Page Source Using driver.execute_javascript In the previous example, we have to comment out (or replace) the driver.page_source line and add the following line: driver.execute_script is a Selenium Python WebDriver API to execute JS in a Selenium environment. Here, we execute a JS script that returns an HTML body element. # pageSource = driver.page_source pageSource = driver.execute_script("return document.body.innerHTML;") The output code looks like this: As you can observe, it only returns the innerHTML of the body element. Like the last output, we do not get the whole page source. To get the entire document, we execute document.documentElement.outerHTML. The execute_script line now looks like this: pageSource = driver.execute_script("return document.documentElement.outerHTML;") This gives us precisely the output we got using the driver.page_source. Fetch Page Source Using Python’s Request Library in Selenium WebDriver This method has nothing to do with Selenium but you can check the “What Is Selenium?” article, it’s a purely Pythonic way to get a webpage source. Here, we use Python’s request library to make a get request to the URL and save the request’s response, i.e., page source to an HTML file and print on the terminal. Here is the script: import requests url = 'https://pynishant.github.io/' pythonResponse = requests.get(url) fileToWrite = open("py_source.html", "w") fileToWrite.write(pythonResponse.text) fileToWrite.close() fileToRead = open("py_source.html", "r") print(fileToRead.read()) fileToRead.close() This method can be used to quickly store a webpage source code without loading the page in the Selenium-controlled browser. Similarly, we can use the urllib Python library to fetch the HTML page source. Get HTML Page Source Using the “view-source” URL This is rarely required, but you can append the target URL with view-source and load it in the browser window to load the source code and save it in manual testing: Programmatically, to take source code of screenshots in Python Selenium (if required), you can load the page using: driver.get("view-source:https://pynishant.github.io/") Get HTML Page Source in Selenium Python WebDriver Using XPath The fourth method to make Selenium WebDriver get a page source is to use XPath for saving it. Here, instead of page_source or executing JavaScript, we identify the source element, i.e., <html> and extract it. Comment out the previous page source fetching logic and replace it with the following: # pageSource = driver.page_source pageSource = driver.find_element_by_xpath("//*").get_attribute("outerHTML") In the above script, we are using a driver method, find_element_by_xpath, to locate the web page’s HTML element. We enter the document using source nod:"//*" and get its “outer HTML,” which is the document itself. The output looks the same as we got earlier using driver.page_source. How To Retrieve HTML Source of WebElement in Selenium To get the HTML source of a WebElement in Selenium WebDriver, we can use the get_attribute method of the Selenium Python WebDriver. First, we grab the HTML WebElement using driver element locator methods like (find_element_by_xpath or find_element_by_css_selector). Next, we apply the get_attribute() method on this grabbed element to get it’s HTML source. Suppose, from pynishant.github.io, and we want to grab and print the source code of the div with id “div1.” The code for this looks like this: from selenium import webdriver driver = webdriver.Chrome() driver.maximize_window() driver.get("https://pynishant.github.io/") elementSource = driver.find_element_by_id("div1").get_attribute("outerHTML") print(elementSource) driver.quit() Here’s the output: Similarly, to get the children or innerHTML of a WebElement: driver.find_element_by_id("some_id_or_selector").get_attribute("innerHTML") There is an alternative way of doing this and achieving same result: elementSource = driver.find_element_by_id("id_selector_as_per_requirement") driver.execute_script("return arguments[0].innerHTML;", elementSource) How To Retrieve JSON Data from an HTML Page Source in Python Selenium WebDriver Modern applications are built with multiple APIs at play. And often, these API dynamically change the content of HTML elements. JSON objects have emerged as an alternative to XML response types. So, it has become essential for a pro Selenium Python tester to handle JSON objects, especially those embedded in <script> HTML tags. Python provides us with an in-built JSON library to experiment with JSON objects. To demonstrate with an example, we load “https://www.cntraveller.in/” in Selenium driver and look-out for SEO schema contained in <script type=”application/ld+json”> </script> to verify that logo URL is included in the “JSON” schema. By the way, if you feel confused, this “SEO schema” is useful to get web pages ranked on google. It has nothing to do with code-logic or testing. We’re using it just for demonstration. We’ll be using LambdaTest for this demo: from selenium import webdriver import json import re username = "hustlewiz247" accessToken = "1BtTGpkzkYeOKJiUdivkWxvmHQppbahpev3DpcSfV460bXq0GC" gridUrl = "hub.lambdatest.com/wd/hub" desired_cap = { 'platform' : "win10", 'browserName' : "chrome", 'version' : "71.0", "resolution": "1024x768", "name": "LambdaTest json object test ", "build": "LambdaTest json object test", "network": True, "video": True, "visual": True, "console": True, } url = "https://"+username+":"+accessToken+"@"+gridUrl print("Initiating remote driver on platform: "+desired_cap["platform"]+" browser: "+desired_cap["browserName"]+" version: "+desired_cap["version"]) driver = webdriver.Remote( desired_capabilities=desired_cap, command_executor= url ) # driver = webdriver.Chrome() driver.maximize_window() driver.get("https://www.cntraveller.in/") jsonSource = driver.find_element_by_xpath("//script[contains(text(),'logo') and contains(@type, 'json')]").get_attribute('text') jsonSource = re.sub(";","",jsonSource) jsonSource = json.loads(jsonSource) if "logo" in jsonSource: print("\n logoURL : " + str(jsonSource["logo"])) else: print("JSON Schema has no logo url.") try: if "telephone" in jsonSource: print(jsonSource["telephone"]) else: print("No Telephone - here is the source code :\n") print(driver.find_element_by_xpath("//script[contains(text(),'logo') and contains(@type, 'json')]").get_attribute('outerHTML')) except Exception as e: print(e) driver.quit() The output contains logoURL and webElement source: Code Breakdown The following three lines import required libraries: Selenium WebDriver, Python’s JSON, and re library to handle JSON objects and use regular expressions: from selenium import webdriver import json import re Next, we configure our script for running it successfully on LambdaTest’s cloud. It took me less than thirty seconds to get started (maybe because I had prior experience with the platform). But even if you are a first-timer, it would take less than one minute. Register on LambdaTest’s official website, login using Google, and click on “Profile” to copy your username and access token: username = "your_username_on_lambdaTest" accessToken = "your lambdaTest access token" gridUrl = "hub.lambdatest.com/wd/hub" desired_cap = { 'platform' : "win10", 'browserName' : "chrome", 'version' : "71.0", "resolution": "1024x768", "name": "LambdaTest json object test ", "build": "LambdaTest json object test", "network": True, "video": True, "visual": True, "console": True, } url = "https://"+username+":"+accessToken+"@"+gridUrl We launch the driver in full-screen mode and load the cntraveller home page with the following line of code: driver = webdriver.Remote( desired_capabilities=desired_cap, command_executor= url ) # driver = webdriver.Chrome() driver.maximize_window() driver.get("https://www.cntraveller.in/") Now, we locate JSON objects containing script using the XPath locator and delete the unnecessary semicolons to load the string in JSON format properly: jsonSource = driver.find_element_by_xpath("//script[contains(text(),'logo') and contains(@type, 'json')]").get_attribute('text') jsonSource = re.sub(";","",jsonSource) jsonSource = json.loads(jsonSource) And then, we check if the logo URL is present. If present, we print it: if "logo" in jsonSource: print("\n logoURL : " + str(jsonSource["logo"])) else: print("JSON Schema has no logo url.") Also, we check if the telephone detail is present. If not, we print the source code of the WebElement: try: if "telephone" in jsonSource: print(jsonSource["telephone"]) else: print("No Telephone - here is the source code :\n") print(driver.find_element_by_xpath("//script[contains(text(),'logo') and contains(@type, 'json')]").get_attribute('outerHTML')) except Exception as e: print(e) Lastly, we quit the driver: driver.quit() How To Get Page Source as XML in Selenium WebDriver If you’re loading an XML-rendered website, you may want to save the XML response. Here’s a working solution for making Selenium get XML page source: drive.execute_script(‘return document.getElementById(“webkit-xml-viewer-source-xml”).innerHTML’) Conclusion You can use any of the above-demonstrated methods and leverage the agility and scalability of LambdaTest Selenium Grid cloud to automate your test processes. It lets you execute your test cases on 3000+ browsers, operating systems, and their versions. Also, you can integrate the automation testing flow with modern CI/CD tools and adhere to the best continuous testing practices. Happy Testing!

When I started working on this post, I had another idea in mind: I wanted to compare the developer experience and performance of Spring Boot and GraalVM with Rust on a demo HTTP API application. Unfortunately, the M1 processor of my MacBook Pro had other ideas. Hence, I changed my initial plan. I'll write about the developer experience of developing the above application in Rust, compared to what I'm used to with Spring Boot. The Sample Application Like every pet project, the application is limited in scope. I designed a simple CRUD HTTP API. Data are stored in PostgreSQL. When one designs an app on the JVM, the first and only design decision is to choose the framework: a couple of years ago, it was Spring Boot. Nowadays, the choice is mostly between Spring Boot, Quarkus, and Micronaut. In many cases, they all rely on the same underlying libraries, e.g., logging or connection pools. Rust is much younger; hence the ecosystem has yet to mature. For every feature, one needs to choose precisely which library to use - or to implement it. Worse, one needs to understand there's such a feature. Here are the ones that I searched for: Reactive database access Database connection pooling Mapping rows to structures Web endpoints JSON serialization Configuration from different sources, e.g., YAML, environment variables, etc. Web Framework The choice of the web framework is the most critical. I've to admit I had no prior clue about such libraries. I looked around and stumbled upon Which Rust web framework to choose in 2022. After reading the post, I decided to follow the conclusion and chose axum: Route requests to handlers with a macro-free API Declaratively parse requests using extractors Simple and predictable error handling model Generate responses with minimal boilerplate Take full advantage of the tower and tower-http ecosystem of middleware, services, and utilities. In particular, the last point is what sets axum apart from other frameworks. axum doesn’t have its own middleware system but instead uses tower::Service. This means axum gets timeouts, tracing, compression, authorization, and more, for free. It also enables you to share middleware with applications written using hyper or tonic. - axum crate documentation axum uses the Tokio asynchronous library underneath. For basic usage, it requires two crates: TOML [dependencies] axum = "0.6" tokio = { version = "1.23", features = ["full"] } axum's router looks very similar to Spring's Kotlin Routes DSL: Rust let app = Router::new() .route("/persons", get(get_all)) //1 .route("/persons/:id", get(get_by_id)) //1//2 async fn get_all() -> Response { ... } async fn get_by_id(Path(id): Path<Uuid>) -> Response { ... } A route is defined by the path and a function reference. A route can have path parameters. axum can infer parameters and bind them. Shared Objects An issue commonly found in software projects is sharing an "object" with others. We established long ago that there were better ideas than sharing global variables. Spring Boot (and similar JVM frameworks) solves it with runtime dependency injection. Objects are created by the framework, stored in a context, and injected into other objects when the application starts. Other frameworks do dependency injection at compile-time, e.g., Dagger 2. Rust has neither runtime nor objects. Configurable dependency injection is not "a thing." But we can create a variable and inject it manually where needed. In Rust, it's a problem because of ownership: Ownership is a set of rules that govern how a Rust program manages memory. All programs have to manage the way they use a computer’s memory while running. Some languages have garbage collection that regularly looks for no-longer-used memory as the program runs; in other languages, the programmer must explicitly allocate and free the memory. Rust uses a third approach: memory is managed through a system of ownership with a set of rules that the compiler checks. If any of the rules are violated, the program won’t compile. None of the features of ownership will slow down your program while it’s running. - "What Is Ownership?" axum provides a dedicated wrapper, the State extractor, to reuse variables across different scopes. Rust struct AppState { //1 ... } impl AppState { fn create() -> Arc<AppState> { //2 Arc::new(AppState { ... }) } } let app_state = AppState::create(); let app = Router::new() .route("/persons", get(get_all)) .with_state(Arc::clone(&app_state)); //3 async fn get_all(State(state): State<Arc<AppState>>) -> Response { //4 ... //5 } Create the struct to be shared. Create a new struct wrapped in an Atomically Reference Counted. Share the reference with all routing functions, e.g., get_all. Pass the state. Use it! Automated JSON Serialization Modern JVM web frameworks automatically serialize objects in JSON before sending. The good thing is that axum does the same. It relies on Serde. First, we add the serde and serde_json crate dependencies: TOML [dependencies] serde = { version = "1.0", features = ["derive"] } serde_json = "1.0" Then, we annotate our struct with the derive(Serialize) macro: Rust #[derive(Serialize)] struct Person { first_name: String, last_name: String, } Finally, we return the struct wrapped in a Json and the HTTP status code in an axum Response. Rust async fn get_test() -> impl IntoResponse { //1 let person = Person { //2 first_name: "John".to_string(), last_name: "Doe".to_string() }; (StatusCode::OK, Json(person)) //3 } The tuple (StatusCode, Json) is automatically converted into a Response. Create the Person. Return the tuple. At runtime, axum automatically serializes the struct in JSON: JSON {"first_name":"Jane","last_name":"Doe"} Database Access For a long time, I used the MySQL database for my demos, but I started to read a lot of good stuff about PostgreSQL and decided to switch. I needed an asynchronous library compatible with Tokio: it's exactly what the tokio_postgres crate does. The problem with the crate is that it creates direct connections to the database. I searched for a connection pool crate and stumbled upon deadpool (sic): Deadpool is a dead simple async pool for connections and objects of any type. - Deadpool Deadpool provides two distinct implementations: An unmanaged pool: The developer has complete control - and responsibility - over the pooled objects' lifecycle. A managed pool: The crate creates and recycles objects as needed. More specialized implementations of the latter cater to different databases or "drivers", e.g., Redis and... tokio-postgres. One can configure Deadpool directly or defer to the config crate it supports. The latter crate allows several alternatives for configuration: Config organizes hierarchical or layered configurations for Rust applications. Config lets you set a set of default parameters and then extend them via merging in configuration from a variety of sources: Environment variables String literals in well-known formats Another Config instance Files: TOML, JSON, YAML, INI, RON, JSON5, and custom ones defined with Format trait Manual, programmatic override (via a .set method on the Config instance) Additionally, Config supports: Live watching and re-reading of configuration files Deep access into the merged configuration via a path syntax Deserialization via serde of the configuration or any subset defined via a path - Crate config To create the base configuration, one needs to create a dedicated structure and use the crate: Rust #[derive(Deserialize)] //1 struct ConfigBuilder { postgres: deadpool_postgres::Config, //2 } impl ConfigBuilder { async fn from_env() -> Result<Self, ConfigError> { //3 Config::builder() .add_source( Environment::with_prefix("POSTGRES") //4 .separator("_") //4 .keep_prefix(true) //5 .try_parsing(true), ) .build()? .try_deserialize() } } let cfg_builder = ConfigBuilder::from_env().await.unwrap(); //6 The Deserialize macro is mandatory. The field must match the environment prefix (see below). The function is async and returns a Result. Read from environment variables whose name starts with POSTGRES_. Keep the prefix in the configuration map. Enjoy! Note that environment variables should conform to what Deadpool's Config expects. Here's my configuration in Docker Compose: Env variable Value POSTGRES_HOST "postgres" POSTGRES_PORT 5432 POSTGRES_USER "postgres" POSTGRES_PASSWORD "root" POSTGRES_DBNAME "app" Once we have initialized the configuration, we can create the pool: Rust struct AppState { pool: Pool, //1 } impl AppState { async fn create() -> Arc<AppState> { //2 let cfg_builder = ConfigBuilder::from_env().await.unwrap(); //3 let pool = cfg_builder //4 .postgres .create_pool( Some(deadpool_postgres::Runtime::Tokio1), tokio_postgres::NoTls, ) .unwrap(); Arc::new(AppState { pool }) //2 } } Wrap the pool in a custom struct. Wrap the struct in an Arc to pass it within an axumState (see above). Get the configuration. Create the pool. Then, we can pass the pool to the routing functions: Rust let app_state = AppState::create().await; //1 let app = Router::new() .route("/persons", get(get_all)) .with_state(Arc::clone(&app_state)); //2 async fn get_all(State(state): State<Arc<AppState>>) -> Response { let client = state.pool.get().await.unwrap(); //3 let rows = client .query("SELECT id, first_name, last_name FROM person", &[]) //4 .await //5 .unwrap(); // //6 } Create the state. Pass the state to the routing functions. Get the pool out of the state, and get the client out of the pool. Create the query. Execute it. Read the row to populate the Response. The last step is to implement the transformation from a Row to a Person. We can do it with the From trait. Rust impl From<&Row> for Person { fn from(row: &Row) -> Self { let first_name: String = row.get("first_name"); let last_name: String = row.get("last_name"); Person { first_name, last_name, } } } let person = row.into(); Docker Build The last step is the building of the application. I want everybody to be able to build, so I used Docker. Here's the Dockerfile: Dockerfile FROM --platform=x86_64 rust:1-slim AS build //1 RUN rustup target add x86_64-unknown-linux-musl //2 RUN apt update && apt install -y musl-tools musl-dev //3 WORKDIR /home COPY Cargo.toml . COPY Cargo.lock . COPY src src RUN --mount=type=cache,target=/home/.cargo \ //4 && cargo build --target x86_64-unknown-linux-musl --release //5 FROM scratch //6 COPY --from=build /home/target/x86_64-unknown-linux-musl/release/rust /app //7 CMD ["/app"] Start from a standard Rust image. Add musl target so we can compile to Alpine Linux. Install the required Alpine dependencies. Cache the dependencies. Build for Alpine Linux. Start from scratch. Add the previously built binary. The final image is 7.56MB. My experience has shown that an equivalent GraalVM native compiled image would be more than 100MB. Conclusion Though it was not my initial plan, I learned about quite a few libraries with this demo app and how they work. More importantly, I've experienced what it is like to develop an app without a framework like Spring Boot. You need to know the following: Available crates for each capability Crate compatibility Version compatibility Last but not least, the documentation of most above crates ranges from average to good. I found axum's to be good; on the other hand, I didn't manage to use Deadpool correctly from the start and had to go through several iterations. The documentation quality of Rust crates is different from crate to crate. All in all, they have room for the potential to reach the level of modern JVM frameworks. Also, the demo app was quite simple. I assume that more advanced features could be more painful. The complete source code for this post can be found on GitHub. To go further: Create an Optimized Rust Alpine Docker Image How to create small Docker images for Rust Using Axum Framework To Create Rest API

A Software Bill of Materials (SBOM) is getting more and more important in the software supply chain. In this blog, you will learn what an SBOM is and how to build the SBOM in an automated way. Enjoy! 1. Introduction An SBOM is a list of software components that makes up a software product. This way, it becomes transparent which software libraries, components, etc., and their versions are used in the software product. As a consequence, you will be able to react more adequately when a security vulnerability is reported. You only need to check the SBOMs for the vulnerable library, and you will know immediately which applications are impacted by the vulnerability. The SBOM of a library or application you want to use can also help you in your decision-making. It will become more common ground that software suppliers will be forced to deliver an up-to-date SBOM with each software delivery. Based on this information, you can make a risk assessment of whether you want to use the library or application. When you are a software supplier, you need to ensure that you deliver an SBOM with each software release. This actually means that you need to create the SBOM in an automated way, preferably in your build pipeline. As written before, the SBOM can help you to check whether a library used in your application contains security vulnerabilities. With the proper tooling, this check can be done in an automated way in your build pipeline. When security vulnerabilities are found, you can fail the build pipeline. One of these tools is grype, which can take an SBOM as input and check whether any components are used with known security vulnerabilities. In a previous post, it is explained how grype can be used. In the post, a Docker image is input to grype, but it is even better to create an SBOM first and then provide it to grype. How to create the SBOM will be explained in this post. When you start reading about SBOMs, you will notice that two standards are commonly used: CycloneDX: an open-source project originated within the OWASP community; SPDX (The Software Package Data Exchange): an international open standard format, also open source and hosted by the Linux foundation. So, which standard to use? This is a difficult question to answer. Generally, it is stated that both standards will continue to exist next to each other, and tools are advised to support both standards. SPDX is initially set up for license management, whereas CycloneDX had its primary focus on security. Reading several resources, the preferred format is CycloneDX when your focus is set on security. Interesting reads are SBOM formats SPDX and CycloneDX compared and the publication Using the Software Bill of Materials for Enhancing Cybersecurity from the National Cyber Security Centre of the Ministry of Justice and Security of the Netherlands. The latter is a must-read. In the remainder of this blog, you will learn how to use syft for building an SBOM. Syft is also a product of Anchore, just like grype is, and therefore integrates well with grype, the vulnerability scanning tool. Syft supports many ecosystems and has several export formats. It definitely supports CycloneDX and SPDX. CycloneDX also has tools for building SBOMs, but Syft is one tool that supports many ecosystems, which is an advantage compared to multiple tools. Sources being used in this post are available at GitHub. 2. Prerequisites The prerequisites needed for this blog are: Basic Linux knowledge; Basic Java knowledge; Basic JavaScript knowledge; Basic Spring Boot knowledge. 3. Application Under Test Before continuing, you need an application to build the SBOM. This is a basic application that consists of a Spring Boot backend and a Vue.js frontend. The application can be built with Maven and contains two Maven modules, one for the backend and one for the front end. More information about the setup can be read in a previous post. It is important, however, to build the application first. This can be done with the following command: Shell $ mvn clean verify 4. Installation Installation of syft can be done by executing the following script: Shell $ curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sudo sh -s -- -b /usr/local/bin Verify the installation by executing the following command: Shell $ syft --version syft 0.64.0 5. Build Backend SBOM Navigate to the backend directory and execute the following command: Shell $ syft dir:. --exclude ./**/sbom.*.json --output cyclonedx-json=sbom.cyclonedx.build-complete-dir.json The parameters will do the following: dir:.: Scan the entire directory in order to find dependencies; –exclude: Exclude already present SBOM files because you want to generate the SBOM file every time anew based on the current state of the repository; –output: Here, you define the output format to use, and you define the file name of the SBOM file. The SBOM file sbom.cyclonedx.build-complete-dir.json is created in the backend directory. Take a closer look at the SBOM format. JSON { "bomFormat": "CycloneDX", "specVersion": "1.4", "serialNumber": "urn:uuid:afbe7b48-b376-40fb-a0d4-6a16fda38a0f", "version": 1, "metadata": { "timestamp": "2023-01-14T16:35:35+01:00", "tools": [ { "vendor": "anchore", "name": "syft", "version": "0.64.0" } ], "component": { "bom-ref": "af63bd4c8601b7f1", "type": "file", "name": "." } }, "components": [ ... ] } The top part consists of metadata: the format of the SBOM, versions used, which tool is being used, etc. The component part consists of a list of all the components syft has found. The complete specification of CycloneDX can be found here. The component list is the following and corresponds to the list of libraries that can be found in the target/backend-0.0.1-SNAPSHOT.jar file. The libraries are located in the directory /BOOT-INF/lib/ in the jar file (the jar file is just a zip file and can be opened with any archive tool). Plain Text backend jackson-annotations jackson-core jackson-databind jackson-datatype-jdk8 jackson-datatype-jsr310 jackson-module-parameter-names jakarta.annotation-api jul-to-slf4j log4j-api log4j-to-slf4j logback-classic logback-core micrometer-commons micrometer-observation slf4j-api snakeyaml spring-aop spring-beans spring-boot spring-boot-autoconfigure spring-boot-jarmode-layertools spring-boot-starter-test spring-boot-starter-web spring-context spring-core spring-expression spring-jcl spring-web spring-webmvc tomcat-embed-core tomcat-embed-el tomcat-embed-websocket Now take a closer look at the jackson-annotations component in the SBOM file. In the properties section, you can see that this component has a property syft:package:foundBy with the value java-cataloger. This means that this component was found in the jar file. JSON { "bom-ref": "pkg:maven/com.fasterxml.jackson.core/jackson-annotations@2.14.1?package-id=9cdc3a1e17ebbb68", "type": "library", "group": "com.fasterxml.jackson.core", "name": "jackson-annotations", "version": "2.14.1", "cpe": "cpe:2.3:a:jackson-annotations:jackson-annotations:2.14.1:*:*:*:*:*:*:*", "purl": "pkg:maven/com.fasterxml.jackson.core/jackson-annotations@2.14.1", "externalReferences": [ { "url": "", "hashes": [ { "alg": "SHA-1", "content": "2a6ad504d591a7903ffdec76b5b7252819a2d162" } ], "type": "build-meta" } ], "properties": [ { "name": "syft:package:foundBy", "value": "java-cataloger" }, { "name": "syft:package:language", "value": "java" }, { "name": "syft:package:metadataType", "value": "JavaMetadata" }, { "name": "syft:package:type", "value": "java-archive" }, ... ] } When you take a look at component spring-boot-starter-web, it mentions that this component was found by java-pom-cataloger. This means that this component was found in the pom file. This is quite interesting because this would mean that syft cannot find transitive dependencies based on the sources only. Execute the following command where the target directory is excluded from the analysis. Shell $ syft dir:. --exclude ./**/sbom.*.json --exclude ./**/target --output cyclonedx-json=sbom.cyclonedx.build-sources.json The result can be found in the file sbom.cyclonedx.build-sources.json and the previously made assumption seems to be right. Only the spring-boot-starter-web and spring-boot-starter-test dependencies are found. This is, after all, not a big issue, but you have to be aware of this. 6. Build Frontend SBOM Navigate to the frontend directory and execute the following command: Shell $ syft dir:. --exclude ./**/sbom.*.json --output cyclonedx-json=sbom.cyclonedx.build-complete-dir.json This analysis takes a bit longer than the backend analysis, but after a few seconds, the sbom.cyclonedx.build-complete-dir.json file is created. Again, similar information can be found in the SBOM. The information is now available from the javascript-lock-cataloger. This means that it originates from the package-lock.json file. Another difference is that the components also contain license information. JSON "components": [ { "bom-ref": "pkg:npm/%40babel/parser@7.20.7?package-id=ca6a526d8a318088", "type": "library", "name": "@babel/parser", "version": "7.20.7", "licenses": [ { "license": { "id": "MIT" } } ], ... License information is included and can be used to check regarding allowed company policies. This information is, however, not yet available for Java packages. 7. Conclusion SBOMs will become more and more important in the software development lifecycle. More and more customers will demand an SBOM, and therefore it is important to automatically generate the SBOM. Syft can help you with that. Besides that, the SBOM can be fed to grype in order to perform a security vulnerability analysis.

As we continue in this series of simulating and troubleshooting performance problems in Scala, now let’s discuss how to simulate the java.lang.OutOfMemoryError: Java Heap space problem. java.lang.OutOfMemoryError: Java Heap space will be thrown by the Scala application when it generates more objects than the maximum configured heap size. Scala OutOfMemoryError Program Here is a sample Scala program, which generates the java.lang.OutOfMemoryError: Java Heap space problem: package com.yc import java.util class OOMApp { } object OOMApp { var myMap = new util.HashMap[String,String]() def main(args: Array[String]): Unit = { var counter = 0; while (true) { System.out.println("Inserting large String") myMap.put("key"+counter, "Large stringgggggggggggggggggggggggggggg" + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg" + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg" + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg" + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg" + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg" + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg" + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg" + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg" + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg" + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg" + "ggggggggggggggggggggggggggggggggggggggggggggggggggggg" + counter) counter = counter+(1); } } } This sample Scala program contains an OOMApp class. Since this class contains the while(true) loop, it keeps on inserting records into the HashMap infinitely. When HashMap grows beyond the maximum heap size (i.e., -Xmx), the java.lang.OutOfMemoryError: Java Heap space will be thrown. The diagram below illustrates the records present in the HashMap. HashMap causing OutOfMemoryError When we executed the program above, as expected, java.lang.OutOfMemoryError: Java heap space was thrown within a few seconds. Even though this is a hypothetical example that simulates java.lang.OutOfMemoryError, this is how a typical memory leak happens in enterprise applications. When records get inserted into a data structure (like HashMap, ArrayList, Set, etc.) and never get removed, java.lang.OutOfMemoryError will be thrown. How To Troubleshoot OutOfMemoryError You can diagnose OutOfMemoryError either through a manual or automated approach. Manual Approach In the manual approach, you will need to capture a heap dump as the first step. A heap dump is a snapshot of memory that will show all the objects in memory, the values contained by those objects, and their references. You can capture a heap dump using one of the 7 approaches given here, but an important criterion is that you need to capture the heap dump right before OutOfMemoryError is thrown. If you are going to capture a heap dump after OutOfMemoryError has occurred, then leaking objects can get garbage collected, and it will become hard (or even impossible) to diagnose the problem. Once heap dumps are captured, you need to import the heap dumps from your production servers to your local machine. From your local machine, you can use heap dump analysis tools like jHat and HeapHero to analyze the heap dumps. Automated Approach On the other hand, you can also use the yCrash open source script, which would capture 360-degree data (GC log, 3 snapshots of thread dump, heap dump, netstat, iostat, vmstat, top, top -H, etc.) from your application stack within a minute and generate a bundle zip file. You can then either manually analyze these artifacts or upload them to the yCrash server for automated analysis. We used the automated approach. Once the captured artifacts were uploaded to the yCrash server, it instantly generated the below root cause analysis report highlighting the source of the problem. Heap dump analysis report generated by the yCrash tool yCrash tool pointing out the root cause of OutOfMemoryError The heap dump analysis report above is from the tool which precisely points out that the HashMap (i.e., myMap) data structure present in the com.yc.OOMApp$.myMap to be the root cause of the memory leak. Also, the tool is reporting that this HashMap is holding 99% of memory. Equipped with this information one can easily go ahead and fix the problematic code. Video To see the visual walk-through of this post, click below: