About 2 years ago I started a small and dirty project – a new catalog for the search engine for the components that JLC PCB provides for their assembly service. The motivation was really personal — I liked the dirt-cheap assembly service, but finding the suitable components for my projects was really painful. So I created JLCParts – a browser-only alternative component catalog that doesn’t need any sophisticated backed. It is just served as a static page.
However, on March 7 2022 I was forced to shut down the service. As of today, March 21 2022 it is back operational and working again. What happened? Who to blame? TLDR: Blame no one; the whole shutdown thing is just a misfortunate result of miscommunication and my service operating in a pretty dark grey area of what is legal. There was no bad faith from JLC PCB nor LCSC, just poor communication. Details below.
How does the service work?
The service downloads a list of available components that JLC PCB provides as a CSV file. However, this data source only contains a text description and no real attributes (e.g., resistance, capacitance, current capabilities, etc). Therefore, we do two things: we try to parse the text description for common passives (e.g., resistors and capacitors), and for the rest, we scrape the attributes from the distributor of the components – LCSC.
Also, the attributes provided by LCSC are a complete mess and they don’t follow a common structure. Therefore, we employ a bunch of heuristics to parse these attributes and normalize them. Internally, we convert every basic quantity into several basic SI units and we also support composite values (e.g., drain to source resistance is given at a certain current). This allows us to present the attributes in a human-readable form, sort them and filter them. We also merge the same attribute values. No more picking “100nf, 0.1µF and 100000 pF”.
Also, in order to keep the service as low-maintenance as possible, the whole thing happens at the user side. The components are stored in IndexedDB. At the time of making the project it was a great choice, however, today it presents several challenges that I will discuss at the end.
What broke?
The update worker stopped working on February 16; it wasn’t able to connect to LCSC server all of the sudden. After some investigation, I found out that LCSC probably blocked the IP address of the worker. I probably triggered some internal mechanism as the day before JLC PCB published about 180 000 new components that the updated wanted to scrape. No wonder why they did that – what I do is against their terms of use and they have every right to do so. Though, I am OK with violating the terms for two reasons – I use the data in good faith (I don’t use them for my own profit; I only bring more customers to them) and actually, my project was supported and promoted by JLC PCB. The first reason is not something that justifies scraping their website, but it was something that lead to the official JLC PCB support, and once I got the support I considered it OK to use the data.
Having the support of JLC PCB and thinking they are pretty much the same company (governed by EasyEDA), I reached out to JLC PCB support with explaining the situation and a kind request to either unblock the worker or explicitly state that they no longer support my service (and I should shut it down). I got no response in over a week. Then I tried other channels. Finally, I got a response that this is something they cannot handle. Therefore, I reached out to LCSC support directly. I started again by explaining the whole thing and my motivation and I again stated that I would like to kindly ask them to unblock the IP address range or state that they don’t wish the service to be operational. I got a brief response in poor English stating that I have to stop immediately. I replied, again, asking whether they even read my whole email and I tried to explain that this service brings customers to them and that it was even promoted by their partner JLC PCB. Well, their response was again brief: stop immediately.
I respected their wish and I shut down the service and let them know. After a few days, it lead to a lot of people asking what happened on social media. And I guess this was the impulse when the support reached out to me again and offered me a solution to making the service work again. Since this is a side project, it took me about a week to properly migrate it to the new data source and make all the adjustments needed for it to work again – however, the service will be quite slow as I will explain in a moment.
Before you make any judgments, I would like to ask you to NOT hate JLC PCB, LCSC, or EasyEDA. In my eyes, this whole shutdown thing is just a misfortunate result of miscommunication, and my service operates in a pretty dark grey area of what is legal. There was no bad faith from JLC PCB nor LCSC, just poor communication. Based on the poor English in the first reply by Judy, I just think that the supported parsed the keywords about using their API and they acted as they were instructed – this is something that shouldn’t be done. And they missed the bigger picture. The follow-up response that offered a solution was written in good English, so I think my requests and context were properly understood. There are a lot of flaws with there are companies, however, on the other hand, you get for what you paid. The situation with JLCParts ended up with a win-win situation, so there is no reason to tell them off.
The Future
When I started this project, JLC PCB was offering about 12 000 components. In this case, the IndexedDb powering the application was working fine. However, as of today, the offer is 66× larger (about 800 000 components). Even the single category of resistors now offers more than 75 000 components. JlcParts wasn’t designed for such large data quantities. I actually never imagined that they would offer more than 50 000 components.
Therefore, especially when searching across categories, the UI might be laggy. And the updates will take quite a long to complete. Also, the project started as featureless, and more and more features were glued in without proper refactoring. On top of that, it was my project that I used to learn React.js. This technical debt is what prevents me from adding some features at the moment (e.g., baking search queries into URL so you can bookmark them).
At the moment I suggest you always perform a full-text search only on as few categories as possible.
I have some ideas to improve this – I would like to use SQLite.js as the data storage. This should drastically improve the search update speeds and as the database will be properly indexed, we could also expect an increase in the data search speed. I would like to join this with a refactoring of the code and adding more convenient features. However, those are the plans, and at the moment I cannot promise you when they will be implemented. Hopefully, soon.
If you have some ideas on how to improve the project feature-wise, feel free to open an issue. If you would like to show me support in maintaining my open source project, consider becoming my GitHub sponsor or Patreon if you prefer so.
Recent news: My open letter to the 3D-printing community
I love the 3D-printing community, but I think there is room for improvement. Let's get better in 2023! Read the full letter.
Support my work!
If you like my work (these blog posts, my software and CAD models) and you would like to see more posts on various topics coming, consider supporting me in various ways:
- You can become my sponsor on Github.
- If you prefer, you can also become my Patreon.
- You can buy me a coffee on Ko-fi,
- or you can buy something from my Tindie store (also see below),
- Or you can just share my work!
If you are interested in knowing what I am up to and recent sneak-peaks, consider following me on social media (Twitter, Instagram, Facebook).
My store offers
I launched new tank cleaning kits for Elegoo Saturn, Saturn S, Mars 1, and Mars 3. You can find them in my store.