Skip to content Skip to sidebar Skip to footer
Home / Safe Number of Memory for Image Uploading

Safe Number of Memory for Image Uploading

Post a Comment

While working on an API that was congenital specifically for mobile clients, I ran into an interesting problem that I couldn't believe I hadn't found before. When working on a Balance API that deals exclusively in JSON payloads, how practice you upload images? Which naturally leads onto the next question, should a JSON API and then brainstorm accepting multipart form information? Is that not going to look weird that for every endpoint, we accept JSON payloads, but then for this one we accept a multipart form? Because we will want to be uploading metadata with the image, we are going to have to read out formdata values too. That seems so 2000's! Then permit's see what we can exercise.

While some of the examples within this post are going to exist using .Internet Core, I recall this actually applies to any language that is beingness used to build a RESTful API. So even if y'all aren't a C# guru, read on!

Initial Thoughts

My initial thoughts sort of boiled down to a couple of points.

  • The API I was working on had been hardcoded in a couple of areas to really exist forcing the whole JSON payload thing. Calculation in the ability to accept formdata/multipart forms would be a piffling scrap of work (and regression testing).
  • We had custom JSON serializers for things like decimal rounding that would somehow manually need to exist washed for form information endpoints if required. Nosotros are even using snake_case equally holding names in the API (dang iOS developers!), which would have to be washed differently in the grade data mail.
  • And finally, is there any way to simply serialize what would take been sent under a multi-part form postal service, and include it in a JSON payload?

What Else Is Out There?

It actually became clear that I didn't know what I was doing. And then like any good developer, I looked to copy. And so I took a look at the public API's of the three social media giants.

Twitter

API Doc : https://developer.twitter.com/en/docs/media/upload-media/api-reference/post-media-upload

Twitter has two different ways to upload files. The first is sort of a "chunked" fashion, which I assume is because y'all can upload some pretty large videos these days. And a more elementary way for just uploading general images, allow's focus on the latter.

It's a multi-part form, but returns JSON. Boo.

The very very interesting part about the API however, is that it allows uploading the actual data in two ways. Either you lot can upload the raw binary data every bit you typically would in a multipart grade postal service, or you could really serialise the file as a Base64 encoded string, and send that as a parameter.

Base64 encoding a file was interesting to me because theoretically (And nosotros we will see afterwards, definitely), we can transport this string information whatsoever fashion we like. I would say that of all the C# SDKs I looked at, I couldn't discover any really using this Base64 method, and so in that location weren't any great examples to go off.

Some other interesting betoken about this API is that yous are uploading "media", then at a later appointment attaching that to an actual object (For example a tweet). So if y'all wanted to tweet out an image, it seems similar you would (correct me if I'm incorrect) upload an image, become the ID returned, so create a tweet object that references that media ID. For my use case, I certainly didn't want to do a two pace procedure similar this.

LinkedIn

API Md : https://programmer.linkedin.com/docs/guide/v2/shares/rich-media-shares#upload

LinkedIn was interesting because it's a pure JSON API. All information POSTs contain JSON payloads, similar to the API I was creating. Wouldn't you guess it, they utilise a multipart class information as well!

Similar to Twitter, they also have this concept of uploading the file first, and attaching information technology to where you lot actually want information technology to end upwardly second. And I totally get that, it's only non what I want to do.

Facebook

API Doc : https://developers.facebook.com/docs/graph-api/photo-uploads

Facebook uses a Graph API. And so while I wanted to take a await at how they did things, and then much of their API is not really relevant in a RESTful globe. They exercise use multi-office forms to upload information, only it's kinda hard to say how or why that is the instance,. Also at this point, I couldn't get my listen off how Twitter did things!

So Where Does That Leave Us?

Well, in a weird way I call up I got what I expected, That multipart forms were well and truly alive. It didn't seem like there was whatsoever great innovation in this area. In some cases, the utilise of multipart forms didn't look so brutal because they didn't need to upload metadata at the same time. Therefore simply sending a file with no attached data didn't look and so out of place in a JSON API. However, I did want to send metadata in the same payload as the image, non take it as a two pace process.

Twitter'southward use of Base64 encoding intrigued me. Information technology seemed like a pretty adept choice for sending data beyond the wire irrespective of how you were formatting the payload. You could ship a Base64 string as JSON, XML or Grade Data and information technology would all be handled the same. It's definitely proof of concept fourth dimension!

Base64 JSON API POC

What nosotros want to do is just test that we tin can upload images equally a Base64 cord, and we don't take whatever major issues within a super simple scenario. Note that these examples are in C# .Cyberspace Core, but again, if you are using any other language information technology should be fairly simple to interpret these.

Start, we need our upload JSON Model. In C# it would exist :

public class UploadCustomerImageModel { 	public string Description { get; set; } 	public string ImageData { get; set; } }

Not a whole lot to information technology. Simply a description field that can be freetext for a user to depict the image they are upload, and an imagedata field that will agree our Base64 string.

For our controller :

[HttpPost("{customerId}/images")] public FileContentResult UploadCustomerImage(int customerId, [FromBody] UploadCustomerImageModel model) { 	//Depending on if you want the byte assortment or a retentivity stream, you can use the below.  	var imageDataByteArray = Convert.FromBase64String(model.ImageData); 	//When creating a stream, you need to reset the position, without information technology yous will meet that you lot always write files with a 0 byte length.  	var imageDataStream = new MemoryStream(imageDataByteArray); 	imageDataStream.Position = 0; 	//Go and do something with the bodily information. 	//_customerImageService.Upload([...]) 	//For the purpose of the demo, we return a file then we can ensure information technology was uploaded correctly.  	//But otherwise you can just return a 204 etc.  	return File(imageDataByteArray, "image/png"); }

Once again, fairly damn simple. Nosotros take in the model, so C# has a great way to catechumen that string into a byte array, or to read it into a memory stream. As well note that every bit we are just building a proof of concept, I echo out the paradigm data to brand sure that information technology'south been received, read, and output like I expect it would, merely not a whole lot else.

Now permit's open up upward postman, our JSON payload is going to look a bit like :

{ 	"description" : "Test upload of an image",  	"imageData" : "/9j[...]" }

I've apparently truncated imagedata down here, simply a super simple tool to turn an image into a Base64 is something like this website here. I would as well notation that when you ship your payload, it should be without the data:image/jpeg;base64, prefix that you sometimes come across with online tools that convert images to strings.

Striking send in Postman and :

Great! And then my image upload worked and the picture of my cat was repeat'd dorsum to me! At this indicate I was actually kinda surprised that it could exist that easy.

Something that became very evident while doing this though, was that the payload size was much larger than the original image. In my case, the image itself is 109KB, merely the Base64 version was 149KB. So near 136% of the original epitome. In having a quick search around, it seems expected that a Base64 version of a file would be about 33% bigger than the original. When it comes to larger files, I think less almost sending 33% more across the wire, but more the fact of reading the file into memory, then converting it into a huge string, and and so writing that out… It could cause a few issues. But for a few basic images, I'thousand comfortable with a 33% increase.

I will also notation that there was a few lawmaking snippets around for using BSON or Protobuf to practice the same thing, and may really cutting down the payload size substantially. The mentality would be the same, a JSON payload with a "stringify'd" file.

Cleaning Up Our Lawmaking Using JSON Converters

Ane matter that I didn't like in our POC was that we are using a cord that almost certainly will be converted to a byte assortment every single time. The great thing near using a JSON library such every bit JSON.net in C#, is that how the client sees the model and how our backend code sees the model doesn't necessarily take to be the exact aforementioned. So let's see if we tin can turn that string into a byte array on an API POST automagically.

First we need to create a "Custom JSON Converter" form. That code looks like :

public class Base64FileJsonConverter : JsonConverter { 	public override bool CanConvert(Type objectType) 	{ 		return objectType == typeof(string); 	} 	 	public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer) 	{ 		return Convert.FromBase64String(reader.Value as string); 	} 	//Because we are never writing out as Base64, nosotros don't demand this.  	public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer) 	{ 		throw new NotImplementedException(); 	} }

Fairly simple, all we are doing is taking a value and converting it from a string into a byte assortment. Also note that we are just worried almost reading JSON payloads here, we don't care nigh writing equally we never write out our image as Base64 (yet).

Adjacent, nosotros had dorsum to our model and we utilize the custom JSON Converter.

public class UploadCustomerImageModel {     public string Description { get; fix; }     [JsonConverter(typeof(Base64FileJsonConverter))]     public byte[] ImageData { get; ready; } }

Note nosotros too change the "type" of our ImageData field to a byte array rather than a string. And so even though our postman test will still send a string, past the fourth dimension information technology actually gets to us, it will be a byte array.

We will likewise demand to change our Controller code as well :

[HttpPost("{customerId}/images")] public FileContentResult UploadCustomerImage(int customerId, [FromBody] UploadCustomerImageModel model) {     //Depending on if you lot want the byte array or a memory stream, you tin can use the below.      //THIS IS NO LONGER NEEDED AS OUR MODEL NOW HAS A BYTE Array     //var imageDataByteArray = Convert.FromBase64String(model.ImageData);     //When creating a stream, y'all need to reset the position, without information technology you will run across that yous e'er write files with a 0 byte length.      var imageDataStream = new MemoryStream(model.ImageData);     imageDataStream.Position = 0;     //Go and do something with the bodily data.     //_customerImageService.Upload([...])     //For the purpose of the demo, we render a file so we can ensure it was uploaded correctly.      //Merely otherwise you can just return a 204 etc.      return File(model.ImageData, "prototype/png"); }

Then information technology becomes even simpler. We no longer need to bother handling the Base64 encoded string anymore, the JSON converter volition handle information technology for united states of america.

And that's it! Sending the verbal same payload will withal work and nosotros accept one less piece of plumbing to do if we make up one's mind to add more endpoints to accept file uploads. Now y'all are probably thinking "Aye but if I add in a new endpoint with a model, I even so need to remember to add the JsonConverter attribute", which is true. But at the same fourth dimension, it ways if in the hereafter you decide to swap to BSON instead of Base64, you aren't going to take to go to a tonne of places and piece of work out how you are handling the incoming strings, it's all in one handy identify.

yorkprourn.blogspot.com

Source: https://dotnetcoretutorials.com/2018/07/21/uploading-images-in-a-pure-json-api/

Post a Comment for "Safe Number of Memory for Image Uploading"