Difference between revisions of "User:Bralani/GSoC2024/Log"
| (16 intermediate revisions by 4 users not shown) | |||
| Line 23: | Line 23: | ||
== Sunday, July 7st, 2024== | == Sunday, July 7st, 2024== | ||
| − | Implemented a grid encoding approach => results are | + | Implemented a grid encoding approach => results are lower than 98.8%. |
== Monday, July 8st, 2024== | == Monday, July 8st, 2024== | ||
| Line 36: | Line 36: | ||
== Friday, July 12st, 2024== | == Friday, July 12st, 2024== | ||
Finished organizing source code. | Finished organizing source code. | ||
| + | |||
| + | == Sunday, July 14st, 2024== | ||
| + | Added acceleration and support for Metal (Mac OS). | ||
| + | |||
| + | == Monday, July 15st, 2024== | ||
| + | Implemented a positional encoding like in NeRF's work -> results improved to 99.3% (accuracy). | ||
| + | |||
| + | == Tuesday, July 16st, 2024== | ||
| + | Implemented an importance sampling approach so as to gather more samples in uncertain areas -> results improved to 99.4% (accuracy) and 99.1% (F1). | ||
| + | |||
| + | == Wednesday, July 17st, 2024== | ||
| + | The main drawback of current NN is that around boundaries it is not so precise. Time to study more complex encodings: I mention useful papers like DeepSDF, Neural Pull, 2D/3D Gaussian Splatting. | ||
| + | |||
| + | == Thursday, July 18st, 2024== | ||
| + | Added vector direction as input feature for the NN. | ||
| + | |||
| + | == Friday, July 19st, 2024== | ||
| + | Tried with SDF neural network, results are not satisfactory because SDF associated to rays are not well defined. | ||
| + | |||
| + | == Saturday, July 20st, 2024== | ||
| + | Reading more in details papers of Gaussians splatting. | ||
| + | |||
| + | == Sunday, July 21st, 2024== | ||
| + | Implemented a first version of 3D Gaussians splatting. | ||
| + | |||
| + | == Tuesday, July 23st, 2024== | ||
| + | Associated a single Gaussian to each positive hit in the training set. The NN must find the largest variance of each Gaussian so as to correctly classify also the negative rays. | ||
| + | |||
| + | == Wednesday, July 24st, 2024== | ||
| + | Continuing coding Gaussian splatting. Had an issue with memory, understood the problem (broadcasting). | ||
| + | |||
| + | == Thursday, July 25st, 2024== | ||
| + | Solved the problem with memory, had good results with 10k rays (better than NIF architecture). However, inference is pretty slow and at the moment it is not possible to train with more rays. Moreover, there are still problems with numbers too big or too small that can break the training. | ||
| + | |||
| + | == Thursday, July 26st, 2024== | ||
| + | Implemented parallelism of Gaussian splatting architecture and inserted double precision floating points. | ||
| + | |||
| + | == Saturday, July 28st, 2024== | ||
| + | Sped up the inference process by taking only the closest gaussians to the ray to predict. (similar tecnique to the grid encoding) | ||
| + | |||
| + | == Monday, July 29st, 2024== | ||
| + | Implemented an adaptive learning technique so as to have a variable number of gaussians during the training process (increase the gaussians in the areas more uncertain and decrease the gaussians whose determinant is very small -> small ellipsoid in 3D). | ||
| + | |||
| + | == Tuesday, July 30st, 2024== | ||
| + | Training with more samples, it is pretty slow. | ||
| + | |||
| + | == Wednesday, July 31st, 2024== | ||
| + | Predicted images are far from ground truth and moreover the inference is very slow. It cannot substitute the ray tracing pipeline. | ||
| + | |||
| + | == Thursday, August 1st, 2024== | ||
| + | Studying an interesting paper of SIGGRAPH 2024 (NVBH): https://weiphil.github.io/portfolio/neural_bvh | ||
| + | |||
| + | == Friday, August 2st, 2024== | ||
| + | Implemented the multi-resolution hash grid. | ||
| + | |||
| + | == Saturday, August 3st, 2024== | ||
| + | Sampled N points along each ray and validated the multi-resolution hash grid with 3 millions ray. Got convergence in few seconds and metric F1 is about 0.985. | ||
| + | |||
| + | == Tuesday, August 6st, 2024== | ||
| + | Got F1 metric on 0.998 with 26 million samples and high resolution hash grid. However, inference is slow so there is the need to implement the BVH approach. | ||
| + | |||
| + | == Thursday, August 8st, 2024== | ||
| + | Implemented a more aggressive sampling part and got 0.9991 as F1 metric. | ||
| + | |||
| + | == Monday, August 12st, 2024== | ||
| + | Recording rendering times for the nbvh network. | ||
| + | |||
| + | == Wednesday, August 14st, 2024== | ||
| + | Implemented a 3D visualizer with frames rendered directly by the NN. | ||
Latest revision as of 15:33, 15 August 2024
Contents
- 1 Development Logs
- 2 Community Bonding Period
- 2.1 Monday, July 1st, 2024
- 2.2 Tuesday, July 2st, 2024
- 2.3 Wednesday, July 3st, 2024
- 2.4 Thursday, July 4st, 2024
- 2.5 Friday, July 5st, 2024
- 2.6 Saturday, July 6st, 2024
- 2.7 Sunday, July 7st, 2024
- 2.8 Monday, July 8st, 2024
- 2.9 Tuesday, July 9st, 2024
- 2.10 Wednesday, July 10st, 2024
- 2.11 Friday, July 12st, 2024
- 2.12 Sunday, July 14st, 2024
- 2.13 Monday, July 15st, 2024
- 2.14 Tuesday, July 16st, 2024
- 2.15 Wednesday, July 17st, 2024
- 2.16 Thursday, July 18st, 2024
- 2.17 Friday, July 19st, 2024
- 2.18 Saturday, July 20st, 2024
- 2.19 Sunday, July 21st, 2024
- 2.20 Tuesday, July 23st, 2024
- 2.21 Wednesday, July 24st, 2024
- 2.22 Thursday, July 25st, 2024
- 2.23 Thursday, July 26st, 2024
- 2.24 Saturday, July 28st, 2024
- 2.25 Monday, July 29st, 2024
- 2.26 Tuesday, July 30st, 2024
- 2.27 Wednesday, July 31st, 2024
- 2.28 Thursday, August 1st, 2024
- 2.29 Friday, August 2st, 2024
- 2.30 Saturday, August 3st, 2024
- 2.31 Tuesday, August 6st, 2024
- 2.32 Thursday, August 8st, 2024
- 2.33 Monday, August 12st, 2024
- 2.34 Wednesday, August 14st, 2024
Development Logs[edit]
Community Bonding Period
- Familiarizing with previous work,especially [Neural Intersection Functions](https://arxiv.org/abs/2306.07191)
Monday, July 1st, 2024[edit]
Downloaded the code from the repo of fall rainy in order to have a common base source code. Then, installed different libraries (like pytorch) to make it work.
Tuesday, July 2st, 2024[edit]
Implemented a different version of the neural network (NIF) to make it work with hit/miss task. Got a 0.99 accuracy with bounding sphere approach in the training set.
Wednesday, July 3st, 2024[edit]
Compared a normal rendering with the neural network (NIF) one: results are still far from being acceptable.
Thursday, July 4st, 2024[edit]
Got an average accuracy of 50% with NIF in predicting the true rendering. Tried with a simple KNN with a billion rays stored, got always 50% of accuracy. -> This means that there are some errors in the pipeline (maybe in generating the rays or conversions in spherical coordinates) because the KNN should be better than the random classifier with a billion rays of training set.
Friday, July 5st, 2024[edit]
Solved issues with rendering, got a maximum accuracy of 98.6% with the true rendering (bounding sphere sampling). Pay attention that this result has been achieved only with a single render.
Saturday, July 6st, 2024[edit]
Implemented a benchmark function that generate different angles of the object (test set). Implemented a different version of loss function (symmetric). Got a maximum accuracy of 98.8% and this result is unbiased -> high confidence.
Sunday, July 7st, 2024[edit]
Implemented a grid encoding approach => results are lower than 98.8%.
Monday, July 8st, 2024[edit]
Tried with different optimizers, results improved to 99.1% with Nesterov Adam that converges faster and it is very stable between epochs.
Tuesday, July 9st, 2024[edit]
Started organizing source code to better understand the workflow.
Wednesday, July 10st, 2024[edit]
Continuing organizing source code.
Friday, July 12st, 2024[edit]
Finished organizing source code.
Sunday, July 14st, 2024[edit]
Added acceleration and support for Metal (Mac OS).
Monday, July 15st, 2024[edit]
Implemented a positional encoding like in NeRF's work -> results improved to 99.3% (accuracy).
Tuesday, July 16st, 2024[edit]
Implemented an importance sampling approach so as to gather more samples in uncertain areas -> results improved to 99.4% (accuracy) and 99.1% (F1).
Wednesday, July 17st, 2024[edit]
The main drawback of current NN is that around boundaries it is not so precise. Time to study more complex encodings: I mention useful papers like DeepSDF, Neural Pull, 2D/3D Gaussian Splatting.
Thursday, July 18st, 2024[edit]
Added vector direction as input feature for the NN.
Friday, July 19st, 2024[edit]
Tried with SDF neural network, results are not satisfactory because SDF associated to rays are not well defined.
Saturday, July 20st, 2024[edit]
Reading more in details papers of Gaussians splatting.
Sunday, July 21st, 2024[edit]
Implemented a first version of 3D Gaussians splatting.
Tuesday, July 23st, 2024[edit]
Associated a single Gaussian to each positive hit in the training set. The NN must find the largest variance of each Gaussian so as to correctly classify also the negative rays.
Wednesday, July 24st, 2024[edit]
Continuing coding Gaussian splatting. Had an issue with memory, understood the problem (broadcasting).
Thursday, July 25st, 2024[edit]
Solved the problem with memory, had good results with 10k rays (better than NIF architecture). However, inference is pretty slow and at the moment it is not possible to train with more rays. Moreover, there are still problems with numbers too big or too small that can break the training.
Thursday, July 26st, 2024[edit]
Implemented parallelism of Gaussian splatting architecture and inserted double precision floating points.
Saturday, July 28st, 2024[edit]
Sped up the inference process by taking only the closest gaussians to the ray to predict. (similar tecnique to the grid encoding)
Monday, July 29st, 2024[edit]
Implemented an adaptive learning technique so as to have a variable number of gaussians during the training process (increase the gaussians in the areas more uncertain and decrease the gaussians whose determinant is very small -> small ellipsoid in 3D).
Tuesday, July 30st, 2024[edit]
Training with more samples, it is pretty slow.
Wednesday, July 31st, 2024[edit]
Predicted images are far from ground truth and moreover the inference is very slow. It cannot substitute the ray tracing pipeline.
Thursday, August 1st, 2024[edit]
Studying an interesting paper of SIGGRAPH 2024 (NVBH): https://weiphil.github.io/portfolio/neural_bvh
Friday, August 2st, 2024[edit]
Implemented the multi-resolution hash grid.
Saturday, August 3st, 2024[edit]
Sampled N points along each ray and validated the multi-resolution hash grid with 3 millions ray. Got convergence in few seconds and metric F1 is about 0.985.
Tuesday, August 6st, 2024[edit]
Got F1 metric on 0.998 with 26 million samples and high resolution hash grid. However, inference is slow so there is the need to implement the BVH approach.
Thursday, August 8st, 2024[edit]
Implemented a more aggressive sampling part and got 0.9991 as F1 metric.
Monday, August 12st, 2024[edit]
Recording rendering times for the nbvh network.
Wednesday, August 14st, 2024[edit]
Implemented a 3D visualizer with frames rendered directly by the NN.
