9 commits

Author	SHA1	Message	Date
Nick Sweeting	f67a5a215a	fix readability indexing process and implement a max total character length on indexed content	2021-04-06 02:01:38 -04:00
jdcaballerov	14df0cbb7c	Update sonic.py ... Sonic buffer accepts 20.000 bytes not unicode characters, since the chunking here is on unicode characters, sending 20.000 characters will overflow sonic's buffer. UTF-8 can take up to 6 bytes, so sending less than (20.000 / 6) rounded minus should be ok.	2021-01-20 14:51:46 -05:00
jdcaballerov	172197ae01	refactor: Remove if LENGTH and use text chunker for every input	2020-12-06 01:14:39 +02:00
JDC	15fbd81480	Change MAX_SONIC_TEXT_LENGTH	2020-12-06 01:14:38 +02:00
JDC	0acf479b70	Partition long strings in chunks for sonic	2020-12-06 01:14:38 +02:00
JDC	f383648ffc	Use a generator for snapshot flush from index	2020-12-06 01:12:47 +02:00
JDC	47daa038eb	Implement flush for search backend after remove command	2020-12-06 01:12:47 +02:00
JDC	c2c01af3ad	Add config for search backend	2020-12-06 01:12:47 +02:00
JDC	5f6673c72c	Implement backend architecture for search engines	2020-12-06 01:12:46 +02:00